MSN Search Slows Over Zealous RSS News Bot
At Search Engine Journal we’ve been noticing some weird increases in our bandwidth lately and perhaps this new announcement by MSN Search has something to do with it. It seems that MSN has been rather active with the spidering of RSS files from various news sites and blogs. Apparently, according to Search Engine Watch and Jeremy Zawondy, MSN is testing a blog and RSS search bot which has “been a tad hyperactive.”
The dark side of RSS is that once a site experiences high levels of RSS based sydication via aggregators, sites, and news bots – site bandwidth becomes a major issue. The more the RSS feed is circulated, the more bandwidth used. If a site is not aware or prepared for such high usage and high transfer of bandwidth, the site may go over their hosting limit or load slowly.
MSN has received some criticism over their over anxious RSS and blog bot and has slowed it down a bit. From the MSN Search blog : “we are crawling RSS and other content that we want to be sure is always fresh in our index. At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot. We have been a bit more zealous in our crawling then we would have liked and we are going to fix that.”
Danny Sullivan points out that MSNBot does have instructions for slowing it down:
When MSNBot is crawling your site, it generally does not try to access your site more often than once every few seconds. If MSNBot determines that your site has a slow connection, it automatically adjusts the frequency. To specify a minimum frequency (in seconds), use the Crawl-delay parameter in the robots.txt file: