Yahoo Slurp is a bot

[teezone]

I'm seeing a steady increase in traffic reported from "Yahoo! Slurp 3.0".. via Inktomi & Yahoo.

Shouldn't this bot be filtered by Statcounter..?

[Arne]

Originally Posted by teezone

I'm seeing a steady increase in traffic reported from "Yahoo! Slurp 3.0".. via Inktomi & Yahoo.

Shouldn't this bot be filtered by Statcounter..?

It is probably a image enabled bot (Inktomi is), as the Google image bot, and they are not filtered.

[teezone]

Not sure I understand..

I'm not seeing any instances of the Google Image Bot in my stats..

[webado]

If you see Google at all then it's the image enabled Googlebot.

[teezone]

I should probably be more clear...

I've been using StatCounter for more than 3 years.. in the past 4 weeks, the Inktomi bot is appearing in stats as a regular visitor, which it's not.

By comparison, I see no instance of a Google bot in the same statcounter reports - from my dom logs, I know Google crawls more than 20k+ pages daily.

This is a new issue - is there any plan to filter it as a bot?

[Arne]

Originally Posted by teezone

I've been using StatCounter for more than 3 years.. in the past 4 weeks, the Inktomi bot is appearing in stats as a regular visitor, which it's not.

What details can you see about the Inktomi visits?
Regular visitors usually are Javascript and images enabled, and use a regular browser.
Image bots are only image enabled and use an "irregular browser", and can't be logged for several of the detailed stats where Javascript is required.

Because they are image enabled they are tracked as a "visitor", by the small (1x1 pixels) image from the Statcounter code.
The "regular" bots are not images nor Javascript enabled so they can't be tracked by any 3rd party tracker like Statcounter.
No filtering by Statcounter is needed to make their visits invisible.

By comparison, I see no instance of a Google bot in the same statcounter reports - from my dom logs, I know Google crawls more than 20k+ pages daily.

So you have not get any visits by the Google Image bot? I can believe that, since it's not often visiting me either. But you will get it eventually.
But even more rarely is the Inktomi visiting me. can't remember when this happened last time.

[rotarysteve]

Got me curious, so I checked out my server logs, and I'm getting crawled from all kinds of bots/spiders. Not seeing "Yahoo! Slurp 3.0".. via Inktomi & Yahoo, but yahoo, googlebot, msnbot, and perhaps a couple dozen more different spiders. Some just look at my robots file and that's it, others give it a good once-over. Interesting.

[webado]

I see a few Inktomi Slurp, a few Googlebot Image and a few others.

Since most of the images on the site I checked are indexed in Google, Googlebot Image must be crawling or has already crawled sufficiently. Might even be crawling under different guises at times.

Regular Googlebot at times may be image enabled without calling itself Googlebot Image.

[teezone]

even more rarely is the Inktomi visiting me. can't remember when this happened last time

I know.. it feels like I woke up in 2004!

What details can you see about the Inktomi visits?

There are two new distinct crawlers appearing as follows (with a variety of IP addresses coming in):

IP Address: Inktomi Corporation (74.6.13.121)
Browser: Yahoo! Slurp 3.0
OS: Unknown
Sunnyvale, California, United States
Inktomi Corporation

IP Address: Yahoo! (98.137.72.226)
Browser: Yahoo! Slurp 3.0
OS: Unknown
Sunnyvale, California, United States
Yahoo!

I see Google Image bot in my domlogs.. but not in the Statcounter reports (which is what I would expect). In apache, it's easy to spot "Googlebot-Image/1.0".

Inktomi now represents about 5% of my "visitors" (90% of my 404s!) - the number has been climbing steadily over the past few weeks, which is why I started this thread.

As a longtime user of Statcounter, I'm pretty obsessed about stats on all my sites (tracking 5), and this is only a recent trend. It seems there is something new going on, the question is what?

Thanks!

[webmaster]

New feature added today.

Block bots from being recorded in your stats.

Check it out here: http://forum.statcounter.com/vb/showthread.php?t=41289