Google has recently created an additional “Only index content meant for Search” section within the spam guide it updated on Oct. 26. The new section advises websites with pages made for direct sharing, collaboration or utility to set a noindex tag or use a robots.txt directive to block crawlers by default for those pages.
According to Google’s changelog this is an addition relating to “blocking indexing of link-shared content.”
The new section includes a link to the introductory page about robots.txt, and according to that introductory page the robots.txt file “is not a mechanism for keeping a web page out of Google.”
What Google Added
Google’s new “Only index content meant for Search” section follows the guide’s advice on spotting spammy accounts and reads:
“If your service lets users create shareable pages, generated outputs, or interactive tools, consider whether that content is meant to be found through search engines or only accessed directly by people with a link.”
“When pages are created for direct sharing, collaboration, or utility rather than public search discovery, apply the noindex rule (using a robots meta tag or X-Robots-Tag HTTP response header) or block crawling in robots.txt by default.”
The guide’s opening paragraph said hosting platforms “may be similarly open to abuse.” The updated sentence adds “tools that let users publish shared pages, generated content, or web apps,” and says spammers may create a large number of “pages or sites” where it used to say sites.
The previous version, last updated Dec. 10, 2025, already suggested noindex on posts from new users until they build a reputation on the platform. That advice is unchanged.
Noindex Or Robots.txt
Google’s robots.txt introduction says a page “that’s disallowed in robots.txt can still be indexed if linked to from other sites.” Google won’t crawl the blocked content in that case, but the URL and possibly the anchor text of links pointing to it can appear in results, without a description.
Google’s page on noindex says the rule only works when robots.txt isn’t blocking the page, because otherwise “the crawler will never see the noindex rule.” Google’s “Control what you share” page, updated Oct. 21, lists robots.txt as applicable to “images and video” and noindex as applicable to “all content types.”
On July 27, I confirmed that Claude’s share path was blocked by a “Disallow” directive in robots.txt. In addition, the share pages also returned a “none” value for the X-Robots-Tag HTTP response header. Google treats “none” as equal to both noindex and nofollow according to its documentation of robots meta tags. I have been unable to determine what Google saw when it initially found those links.
New Search Console Checks
The new spam guide from Google states you should be verifying your website or subdomain through Search Console for your platform to receive emails when it gets a manual action or has a security issue. It further suggests you check the Page Indexing report and Search performance report on an ongoing basis for “unexpected spikes” of indexed pages, impressions or queries which do not align with your platform’s intent.
A new bullet under the guide’s monitoring advice suggests tracking creation and indexing at the platform level, with alerts for spikes in newly published public pages or “a small number of users generating a large share of indexable URLs.”
Why This Matters
Google’s changelog says its Oct. 26 update to the spam guide aims to help platforms “keep user-created pages that aren’t meant for Search out of the index and detect spam earlier.”
For share links, generated outputs, and app pages, check that any noindex can reach Googlebot. A robots.txt rule that blocks Googlebot from those pages means Google never reads it.
Looking Ahead
The Page Indexing report that Google’s new monitoring advice points to has a status called “Indexed, though blocked by robots.txt.” Google’s help for that status says to remove the block and use noindex. Share URLs that Google indexes despite a robots.txt block can appear under that status in Search Console.
Featured Image: Thaspol Sangsee/Shutterstock
