Google Makes Change To Webmaster Guidelines: Don’t Block Ads With Robots.txt

SMS Text

An update has been made to the technical guidelines section of Google’s Webmaster Guidelines that specifically states site owners should not block the Google ads destination URL with the robots.txt file. This information was first reported by Barry Schwartz of Search Engine Roundtable.

The new text reads:

Make efforts to ensure that a robots.txt file does not block a destination URL for a Google Ad product. Adding such a block can disable or disadvantage the Ad.

It’s interesting that this change is made just days after Matt Cutts announced a change to the page layout algorithm that penalizes pages with too many ads above the fold. This is just pure speculation on my part, but maybe Google is trying to be proactive in thwarting sites that try to block the presence of ads with their robots.txt file.

For example, if a webmaster was unhappy with this new algorithm change and wanted to keep the the top of their page full of ads, they made try to hide the ads from Google by blocking them in the robots.txt file so they can’t be crawled.

In doing so they may think they’re avoiding a penalty, but with this new Webmaster Guidelines change it appears they will be doing more harm than good since they will be effectively disabling the ad all together.

Or maybe I’m reading this all completely wrong. What do you think is the reason behind this recent change to Google’s Webmaster guidelines? I’d love to hear your thoughts!

Editor’s Note: This post has been amended to include the source of the first report. According to Search Engine Roundtable, these guidelines have now been removed.


Matt Southern
Matt Southern is the lead news writer at Search Engine Journal. His passion for helping people in all aspects of online marketing flows through in... Read Full Bio
Matt Southern
Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • You’re reading it wrong. As I mentioned over on, “destination URL for a Google Ad product” is an AdWords landing page, which they need to be able to spider, whereas the paragraph underneath that has to do with links that pass PageRank.

  • Making a Guidelines change like that could indicate that Google is actually seeing the impact that robots.txt crawl disallows are now actually having on their revenue numbers. Practically speaking, however, if the ads are “disadvantaged” by disallow commands, the crawler can’t do its job of serving the most relevant ads. If you still want AdSense income from the page, don’t disallow it from being crawled.

  • I don’t think this is an SEO change at all, but has to do with quality scores in your AdWords account, but maybe I’m just reading this wrong as well.

  • Great Post!! why google changes time to time…. some time that changes are so dangerous for a website…

  • confusion .. I believe we have given Google too much importance to Google and we thought too good company .. Google is actually a money machine and must protect its also going against their own rules that require us to industry experts.

  • One has to “Make reasonable efforts to ensure that advertisements do not affect search engine rankings”:
    Use the nofollow or the redirect methods to ensure your ads are not affecting ranking.

  • Hello,

    I can’t actually see that change in the text, so it’s either a mistake or they updated/released the change before they were ready?

    It does seem like such a dumb move as looking at my GWT I see a whole bunch of these types of pages still showing up in my error reports, so until Google knows how to not crawl and index these types of pages I think best practice would say you block them from being crawled with Robots.txt

  • Hi,

    Nice Post, But i don’t Understood This line ” Google is trying to be proactive in thwarting site “. Can you please tell me whats does it mean?


  • Thwarting refers to preventing. I guess Google is proactive in Preventing your website from Great outcome…! Don’t blame me if I’m wrong…..

  • Great post but I think that the backlink according to google is more important. Also think that the unique article is more essential than all. Again thanks for the article.

  • Thanks for the post. But I don’t understanding why Google changes time to time it is very difficult for the websites and SEO as well?????

  • pankaj

    Is the user have a problems with too many ads, if not i don’t think so it should be penalized , Google should penalizes the sites only if there are irrelevant ads on page.

  • Hi,

    I don’t think so these changes are made by Google and if yes, then why Google changes its norms so quickly. According to me , Google should penalise the websites with non- relevant ads.

    • yes Arcadian agree with your comments