Linux Archive

I’ve just had to do some Apache Log analysis for the various Find Locally domains. As all the sites are run off one server, the log files contain data from all the domains. This makes it a little difficult to get any meaningful information without having to analise each domain one at a time, but before getting to that, I’d have to get all the data into a format I could do something with.

After downloading the 140 Mb log files, I used M$ Access to import the delimited files and then used the MySQL ODBC Connector to export the tables into a MySQL database on my laptop (it’s a Core Duo and pretty much the fastest machine I have).

I then decided to get rid of the Bot traffic from the 5 million hits so I could be left with something usable. As Apache only stored the IP address of the remote machine, I sent PHP on a 12 hour mission to use gethostbyaddr() to store all the remote domains. This made it a lot easier to filter out known Search Engines and the like.

Having gotten rid of all the junk data and having it in a decent database, it made it easy to perform analysis on the remaining logs. As our domain names are all in the same format, using PHP made it easy to dump out a few tables of results showing exactly what I need. All I need now is to get it into a set of automated scripts so I don’t have to do it by hand next month.

Phew, it took me three goes to get that posted! After moving to the new Root Server I was having problems with some of the scripts that automatically create files somewhere in the document root.

To get this working I just decided to run Apache under the same user as the owner of the document root. This then caused the problem that all the session variables were not being saved! Quick chown to the new user on the temp directory used to save the session files sorted that.

After weeks of planning, massive reconfigurations, new backup scripts and file moves, I think I´m finally ready to move servers!
 
I´ve just to figure out a night when we should be fairly quiet to do the move so the sites aren´t down too long. I´ve got it all planned out and should be able to do it in stages so that I don´t have to do a series of all-nighters. Plus it is the holiday season so the distruption to the public should be minimal.
 

Just got to wait until Tuesday for a test script to run successfully and then I can go ahead and do it!

Well, I´ve got around to setting up the development servers in the office, distributing the development files and databases across them and set up the backup routines. I´ve also changes all the files to take into account the server move so all I need to do now is set up the new servers, move accross and test.

 
The move and testing are pretty simple since it´ll just require taking down the database at a certain point, moving the files accross and bringing it live again. As long as I do this at stupid-o-clock then nobody should loose any info or even notice.
 
The main problem is the DNS shift. If I was moving to another host then this would take 24 hours and result in serious downtime, but since the same DNS servers are being used and there is just an IP address change it should go through a lot quicker. I´ll still do it at night to minimise disruption though. The only problem is that it requires going through a two minute process to change the IP address the domain points to in the control panel and we´ve got about 4,000 domains! I think I might need a sleeping bag and a lot of coffee that night!