WebmasterWorld Bans Search Engine Bots from Crawling
Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; “We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots – but it is becoming an increasingly difficult problem to control.”
Members are worried about how they can search WebmasterWorld, since the internal search engine is sub-par. Brett hopes that they can allow the bots again within 60 days, but it depends. Best of luck Brett!
Forum discussion at WebmasterWorld.
Barry Schwartz is the Editor of Search Engine Roundtable and President of RustyBrick, Inc., a Web services firm specializing in customized online technology that helps companies decrease costs and increase sales.