WebmasterWorld Bans Search Engine Bots from Crawling

SMS Text

WebmasterWorld Bans Search Engine Bots from Crawling

Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three…

Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; “We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots – but it is becoming an increasingly difficult problem to control.”

Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!

Forum discussion at WebmasterWorld.

Barry Schwartz is the Editor of Search Engine Roundtable and President of RustyBrick, Inc., a Web services firm specializing in customized online technology that helps companies decrease costs and increase sales.

Download: The Beginner's Guide to SEO
A Complete Guide to Getting Started in SEO.