Yahoo Slurp is a bot

Discussion in 'Help' started by teezone, Jun 7, 2012.

Thread Status:
Not open for further replies.
  1. teezone

    teezone New Member

    Joined:
    Feb 8, 2006
    Messages:
    67
    Likes Received:
    0
    I'm seeing a steady increase in traffic reported from "Yahoo! Slurp 3.0".. via Inktomi & Yahoo.

    Shouldn't this bot be filtered by Statcounter..?
     
  2. Arne

    Arne New Member

    Joined:
    Sep 25, 2004
    Messages:
    5,464
    Likes Received:
    0
    It is probably a image enabled bot (Inktomi is), as the Google image bot, and they are not filtered.
     
  3. teezone

    teezone New Member

    Joined:
    Feb 8, 2006
    Messages:
    67
    Likes Received:
    0
    Not sure I understand..

    I'm not seeing any instances of the Google Image Bot in my stats..
     
  4. webado

    webado Moderator

    Joined:
    Apr 29, 2004
    Messages:
    28,159
    Likes Received:
    1
    If you see Google at all then it's the image enabled Googlebot.
     
  5. teezone

    teezone New Member

    Joined:
    Feb 8, 2006
    Messages:
    67
    Likes Received:
    0
    I should probably be more clear...

    I've been using StatCounter for more than 3 years.. in the past 4 weeks, the Inktomi bot is appearing in stats as a regular visitor, which it's not.

    By comparison, I see no instance of a Google bot in the same statcounter reports - from my dom logs, I know Google crawls more than 20k+ pages daily.

    This is a new issue - is there any plan to filter it as a bot?
     
  6. Arne

    Arne New Member

    Joined:
    Sep 25, 2004
    Messages:
    5,464
    Likes Received:
    0
    What details can you see about the Inktomi visits?
    Regular visitors usually are Javascript and images enabled, and use a regular browser.
    Image bots are only image enabled and use an "irregular browser", and can't be logged for several of the detailed stats where Javascript is required.

    Because they are image enabled they are tracked as a "visitor", by the small (1x1 pixels) image from the Statcounter code.
    The "regular" bots are not images nor Javascript enabled so they can't be tracked by any 3rd party tracker like Statcounter.
    No filtering by Statcounter is needed to make their visits invisible.

    So you have not get any visits by the Google Image bot? I can believe that, since it's not often visiting me either. But you will get it eventually.
    But even more rarely is the Inktomi visiting me. can't remember when this happened last time.
     
  7. rotarysteve

    rotarysteve New Member

    Joined:
    Aug 27, 2005
    Messages:
    1,885
    Likes Received:
    0
    Got me curious, so I checked out my server logs, and I'm getting crawled from all kinds of bots/spiders. Not seeing "Yahoo! Slurp 3.0".. via Inktomi & Yahoo, but yahoo, googlebot, msnbot, and perhaps a couple dozen more different spiders. Some just look at my robots file and that's it, others give it a good once-over. Interesting.
     
  8. webado

    webado Moderator

    Joined:
    Apr 29, 2004
    Messages:
    28,159
    Likes Received:
    1
    I see a few Inktomi Slurp, a few Googlebot Image and a few others.

    Since most of the images on the site I checked are indexed in Google, Googlebot Image must be crawling or has already crawled sufficiently. Might even be crawling under different guises at times.

    Regular Googlebot at times may be image enabled without calling itself Googlebot Image.
     
  9. teezone

    teezone New Member

    Joined:
    Feb 8, 2006
    Messages:
    67
    Likes Received:
    0
    I know.. it feels like I woke up in 2004!

    There are two new distinct crawlers appearing as follows (with a variety of IP addresses coming in):

    IP Address: Inktomi Corporation (74.6.13.121)
    Browser: Yahoo! Slurp 3.0
    OS: Unknown
    Sunnyvale, California, United States
    Inktomi Corporation

    IP Address: Yahoo! (98.137.72.226)
    Browser: Yahoo! Slurp 3.0
    OS: Unknown
    Sunnyvale, California, United States
    Yahoo!

    I see Google Image bot in my domlogs.. but not in the Statcounter reports (which is what I would expect). In apache, it's easy to spot "Googlebot-Image/1.0".

    Inktomi now represents about 5% of my "visitors" (90% of my 404s!) - the number has been climbing steadily over the past few weeks, which is why I started this thread.

    As a longtime user of Statcounter, I'm pretty obsessed about stats on all my sites (tracking 5), and this is only a recent trend. It seems there is something new going on, the question is what?

    Thanks!
     
  10. webmaster

    webmaster StatCounter CEO Staff Member

    Joined:
    Mar 8, 2003
    Messages:
    3,242
    Likes Received:
    0
  11. teezone

    teezone New Member

    Joined:
    Feb 8, 2006
    Messages:
    67
    Likes Received:
    0
    This is great news, thanks for the tweak!
     
  12. devbalic

    devbalic New Member

    Joined:
    Jun 6, 2012
    Messages:
    10
    Likes Received:
    0
    Even i found some Google image bot visits in my stat counter as my website has very targeted images and suddenly my visits had gone down.. i wondering how suddenly my visits have gone down then i found that this Google image bots have not crawl my site for some instance. i want to filter this one but we can not filter them :(
     
Thread Status:
Not open for further replies.

Share This Page