In The Behavior of the Yahoo! Bots - Part I, I talked about an in-depth analysis of visits from Yahoo! domains to eKstreme.com in April. That first post talked about the user agents used. This post will discuss how the bots accessed the pages in more detail.

Yahoo! Slurp

The first thing I looked at is which remote hosts Slurp used. There was a simple pattern: there were three 'domain series' that requested pages while identifying themselves as Yahoo! Slurp:

In all cases, the * was a number. I could not see a pattern to assign function to each series. Remember, that these are not where the majority of Slurp hits came from: Tens of thousands of requests came from *.inktomisearch.com, which seems to be the main Slurp domain.

Slurp also has an annoying behavior: I found many example when Slurp would request the same page multiple times (2-7) within a 5-10 second time span. I even found examples of multiple requests per second. I wonder why Slurp does that. Is Yahoo! trying to gauge how good a site is under stress? Is it a buggy Slurp? I don't know.

Slurpy Verifier

This one is new to me. It looks likes a bot: no referring links, same IP address for each request (rdev25.yst.corp.yahoo.com), and, requesting only a single page off this blog. Others have observed identical behavior (see yeraze.com, The Hive Archive, and Ham Radio Blog). It hasn't done anything particularly annoying (or useful, for that matter) so far, but we should keep an eye on it for future reference.

Scooter

Scooter was Altavista's bot, and it seems it's still alive. It requested pages four times, three of which were for the guestbook. I wonder if it's scraping links or emails ;).

One notable feature of Scooter is that it sets the HTTP_REFERER header to be equal to the page it's requesting. I'm seeing more bots send a referring link header lately, so maybe this is a trend that's emerging.

Conclusions

This pretty much sums it up. The Yahoo! bots are quite active and diverse. Given the recent Yahoo! index updates (after months of stagnation), it's refreshing to see Yahoo! take such an interest in websites. I actually decided to look at the Yahoo! bots since Yahoo! started sending me significant traffic; only a few months ago, Yahoo! couldn't have cared less about eKstreme.com.

And in closing: if anyone has any more info about any of those bots, please drop me a line.

Technorati Tags: Yahoo, Slurp, Slurpy Verifier, bots, robots, search engines