Live.com Spambot Ignores robots.txt
Oh, MSNbot, when will you ever learn? I won't rehash the story that lead me to blocking MSN's referral-spamming bot, and that seems to have worked a bit. The problem is that the referral spam is still coming in! Yes, MSNbot is blocked but the spammy hits are still coming in.
Case in point, this hit from today over at Social Alerter:
/tips/how-not-get-dugg
- At: 19 April 2008 11:04:39 AM GMT
- Referred from: http://search.live.com/results.aspx?q=alerts&mrt=en-us&FORM=LIVSOP
- Remote: livebot-65-55-165-107.search.live.com (65.55.165.107)
- Request: HTTP/1.0 GET
- Accepting:
- HTTP: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
- Charset:
- Enconding:
- Languages: en-us
- UA: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 1.1.4322)
- Cookies:
Is it just me or is this beyond comical now?
Subscribe to Things of Sorts
If you liked this post, please subscribe to the Things of Sorts RSS feed: 
April 20th, 2008 at 4:23 am
I’m confused… if you blocked it, then how is it coming in? Or do you mean somethung different when you say you blocked it? I did it by IP block, thusly:
SetEnvIfNoCase Remote_Addr “(65\.55\.165).*” bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
April 20th, 2008 at 9:47 am
Hi Michael,
I blocked it via robots.txt. I’m instating an IP-based block because clearly gently suggesting to the bot to piss off does not work.
Pierre