Reading Server Statistic logs like AWSTATS

[JWJ]

I'm sure everyone already has their own favourite way of reading those 'raw access logs' that we get from the likes of awstats but I thought this worth a mention in case anyone out there is struggling.

I have recently discovered "Auto Filtering" in Excel and it's a fantastic tool for analysing the stats. I download a raw log to my PC, unzip it and open it in Excel. At this point Excel provides 'import' options and by selecting 'Delimited' and on the Next screen 'Space', Excel cleverly sets out the log in columns and rows.

Auto Filter (accessed from the Data Menu) should then be selected and it will place a drop down selection list on row 1. It is then possible to drop down the list for any of the columns and select an item. Once selected all the rows that do not contain that term get hidden from view.

If you want to follow a particular IP address then just select that address in the first column. If you want to examine the hits to a particular page just select that page in the appropriate column. To look at all 404's just .... oh, you get the idea.

One trick that's really cool is to select one criteria .... IP, error code, page, etc .... then change the background colour of just those rows. When you take away the selection you can easily see the highlighted rows amongst all the others.

The selection boxes also allow for custom selections so you can key in your own, like "beginning with 123.45." to pick up on a range of addresses or "containing abcde", where abcde is any string or "ending with .php" to see a particular file type. The possibilities are endless.

[webado]

Great. I should give this a try. I'm very rusty as Excel though

I usually just ignore AWSTATS since it's pretty useless. And writing regular expressions there to filter stuff is very frustrating. Excel sounds like a much better option. Thanks for sharing this.

[activewebs]

Hi Guys
Hey Thanks John , ive just tried that and it works , Ive never realy been able to sort how log files work or how to show the data in a way that means somthing

Now ive got an Excel spread sheet full of data that I dont understand and worries me cos thers loads of stuff like 404 and robot.txt , we never set a robot text !!!, can any one help ?

Rich

[RockyB]

Weblog Expert works like a charm for me. I'm only using the free version though, so I can't get access to all the fancy features :P

[JWJ]

Originally Posted by activewebs

we never set a robot text !!!, can any one help ?

Rich,

Googling for robots.txt will bring you up a wealth of tutorials. As you probably know robots.txt can do quite a lot by way of controlling what the bots can crawl on your site but if all you want to do is prevent the 404 errors (and it is good practice to have a robots.txt file anyway) then the simplest form is this. It simply grants permission to all bots to crawl everything.

User-agent: *
Disallow:

Just put those two line in a file using Notepad, save it as robots.txt and upload it to your root directory.

[activewebs]

Hi Guys
Thanks for that John , I'l have a look at the stuff and see if it makes any sense ,
To be honest ( admiting to being dumb! ) we dont realy know what sever logs are, is it a list of what the server has been , err , well ,, serving ?

I put that stuff in to excell , and loads of fields show up , i think some of them tell me what pages are being loaded , so - yep I can see the page titles of the site and also there are 404's ( not many ) and a robot.txt , now we are on windows hosting and if we look in our root folder thers nowt there !! except the folders to put stuff in , no sign of a robot txt.
And when we look at the excel sheet (scary) theres loads of stuff that we dont understand , is there some log file tool ( free please ) that will help ??

Thanks
Rich

[webado]

Normally your webhosting would provide a log viewer tool in your control panel. Something like AWSTATS or Webalizer.

What you see in the log is every single access request that was sent to and received by your server on your account.

[rotarysteve]

That is interesting, I would like to see something like that on my site. I use FTP only and never been set up with a control panel. Never seen any extra folders either.

Things that make me go hmmmmmm

Steve

[JWJ]

Rich,

My suggestion about Excel and it's AutoFiltering tool was aimed more at analysing the logs and examining particular entries. It's not a good way of viewing what hits you've been getting because, as you've found out, raw logs are not very user-friendly. If you want to follow a particular IP or pull out all the 404's or find the GET-APPI's then it's great.

Daniel recommended Weblog Expert ... have a look at that.

[activewebs]

Dohh

Stats analysin costs a bit more from our host , we can get the logs via ftp but , thats all , this running em thro excell is the best we have ever done !

Rich