May 17 2007

DreamHost Blocks Googlebot on Client Sites

According to the DreamHost site, they host over 500,000 ‘domains’ (not sure if they mean parked domains and sites, but that’s probably the case).

Email text:

This email is to inform you that a few of your sites were getting hammered by Google bot. This was causing a heavy load on the webserver, and in turn affecting other customers on your shared server. In order to maintain stability on the webserver, I was forced to block Google bot via the .htaccess file.

[Limit GET HEAD POST
order allow,deny
deny from 66.249
allow from all]

You also want to consider making your files be unsearchable by robots and crawlers, as that usually contributes to high number of hits. If they hit a dynamic file, like php, it can cause high memory usage and consequently high load…

I do not think I have heard of a more ridiculous approach by a web server company to ‘help’ its customers by manually altering their .htaccess file to block Google. Furthermore, to ask their customers to block other search bots is completely off the hook.

It would be much more cost effective for the customer to choose a hosting service which can handle the dreaded Googlebot, instead of signing their own incoming traffic death certificate by blocking Google, and all of the incoming traffic which comes as a result of being indexed.

[Hattip to SEOpedia]

Written By:

Loren Baker

More Posts By Loren Baker

According to an email sent to the owner of a Romanian site, Zoso.ro (reminds me of Led Zepplin), the DreamHost web hosting & server company is demanding that its high traffic sites block the Googlebot. According to the DreamHost site, they host over 500,000 'domains' (not sure if they mean parked domains and sites, but that's probably the case). Email text: This…

Jeff Avery

I tend to agree here. I have been using Dreamhost now for about a year and they have been great. I am transferring Domains over from 1and1 and I have had no issues.

To me, the point made is valid. If you are on the shared plan (10.00 per month with all the amenities) and you are expecting dedicated server results you are a bit off.

I would agree modifying your code is not the answer but if you are going down the high volume road you should probably consider paying for the right tool for the job.....just my two cents.

-Jeff

Jay

I've been their before but because of some problems I transfer my clients to 1&1 webhost. They offer large webspace and database. I love their support and they're very helpful (it depends of what problem you have). Try to visit their website http://www.1and1.com/?k_id=19590282

celebrity

celebrities

Josh

I recently got a Dreamhost account for the Jabber server. Blocking Googlebot goes too far though. I would feel like my site was being molested. Thanks for the warning.

Hellhound

If you look over at their forums, a bunch of users seems to be quite happy with being promised huge bandwidth and storage, although they should never ask for actually getting it.

Oh, and in their blog they have posted a, uhm, "justification" for overselling:

http://blog.dreamhost.com/2006/05/18/the-truth-...

Yeah, everybody oversells... but eventually has to get the hardware to satisfy the demand.

Heah Dreamhost, don't offer it if you don't have the resources.

What a bunch of dimwits.

Cvos SEO

While I do have pity on the Dreamhost server admins, blocking Googlebot is not the answer. Yes its true that Dreamhost has hundreds of thousands of accounts, many of which contain spam, but they need to throttle googlebot, not block it. Using webmaster central can reduce bot hits.

Blocking Google is like turning your website off at 5pm when you leave your office. No e-commerce business would ever dream of this.

Form builder

great post, some of our users ask as about this issue and that they don't understand how come our forms are indexed and that the rest of the site is not...
well...thats becuase of a simpe reason.,.forms are not being hosted on the same user domain..
keep updating us!

Hong Xiaowan

That is too bad, without Googlebot, what is the value of a website?

Pozycjonowanie

It's really strange i think sometimes it's good to do this (when someone don't tel googlebot to go back after 2-3 days). But when they host over 500,000 ‘domains’ - it will block everyones website from google. I agree with Siropei "The best thing he can do is move to a more reliable provider :)" when more users go to other providers maybe they will think more when making plans to do something like block googlebot. Greetings

Jack Web

I think Dream host was suppose to inform the client about the issue. He should have asked to upgrade a sever. Directly blocking google bots without informing the client initially was not so good method.

Arpit

"Ohh Nimo, you are a small little Fish. Let me take you out of water to save you from crocodile."

This is their approach to help their clients. . LOL
:D:D

Pchere

Dreamhost edited my .htaccess too to prevent crawling by Googlebot. Then they misconfigured it to bring the site down. Later they apologized saying it was an error on their part.

Michael Martinez

There is too little information for anyone to do a proper analysis. With a dedicated server, I have occasionally had to tell Google and Yahoo! to stay out of my domains for as much as a couple of weeks. They can really hammer a server when they do a deep-crawl, which doesn't happen often.

A site that is getting deep-crawled may be too busy for shared hosting. The last shared hosting provider I had was kind enough to give me six weeks to find a dedicated plan once I'd outgrown their resources.

I've known people whose hosting providers merely shut down their sites without so much as a heads up email saying, "You should get a new hosting provider immediately".

It sounds very much like growing pains for a successful site. Maybe it's just time for them to take the next step and move to dedicated hosting.

Cristian Mezei

A support representative from DreamHost just replied in my blog post.

have a look there guys.

Their "view" is ... how should I say ..

Lucas Castro

the site's owner cannot just go to webmaster tools at google and set google bot to slow down?

as far as i know, there is an option to reduce crawling timeout...

John

This is not the first time that something like this has happened and found out (I wonder how many times it happens and nobody figures out why the site is having a hard time in Google). http://groups.google.com/group/google-sitemaps/...

greg

spenser - only suckers pay $10/month. You can find a coupon in about 30 seconds of searching for $97 off their service. Plus, with their referral plans my hosting is now completely free until about October this year.

As has been stated before, this is a site-specific block of the crawlers and is only necessary for sites with inefficiently written code. Run a decent system with decent caching and it won't be a problem unless you run a big community (and if you run a big community, monetize it and you can afford a dedicated machine...)

I got the letter once for one of my sites. They put it into my .htaccess so once I installed a cache I removed the block. Problem solved.

Greg

spenser

Greg,

If you host with dreamhost for the space, bandwidth, etc, but do not care about traffic, then perhaps it is a great deal for you. Ten bucks for lunch? Well, ok.

For anyone else who is trying to build traffic, having spiders blocked would be self defeating to the point of making even ten dollars a bad investment.

.

greg

I'm on dreamhost, I got this letter, and I stick with them. Sorry, but how can you complain when the host is less per month than I spend on lunch AND they provide shell and unlimited svn repositories?

Blah blah, spend $10 on a better host. Find one that offers me svn and jabber and all the other services for the same amount of money and I'll switch.

jfj3rd

This has got to go down in history as the lamest possible hosting blooper known to man.

Confirmed fact or fake this is going to get a lot of attention from the SEO community and therefore is a great post.

Thanks for making my day, I only wish I found it first!

Sushubh

freaking awesome news. gotta be the funniest thing i have heard today...

i have had bad experience dealing with bad hosts which modified some code on my site without informing me which was causing too much server load.

but this is ridiculous.

Loren Baker, Editor

@Mike : Yes, this has been confirmed, not directly from DreamHost but via a DigitalPoint thread which shows they've been practicing this searchbot blocking for some time now.

Siropel

They are OK unless you have a busy site / blog. They don't have problems with the bandwidth but they will kind of cut you of for resources.
Go with a real provider than doesn't give you the stars for 10$ :)

Mike Bogo

Hmm, weird - is this confirmed?

I use dreamhosts for some of my sites, and they've been good for the most part.

Tomche

This is indeed as ridiculous as it can get. If I was their customer, I would hav bid good bye by now.

SEO Expert Blog

This looks very much like a hoax.

Loren Baker, Editor

@Steven: LOL, yeah, that would sound like the next step according to this company's line of logic.

Steven Bradley

Let me guess. The next email will look something like:

This email is to inform you that a few of your sites were receiving a lot of traffic. This was causing a heavy load on the webserver, and in turn affecting other customers on your shared server. In order to maintain stability on the webserver, I was forced to block all incoming traffic to your site.