1. SEJ
  2.  ⋅ 
  3. SEO

Google Adds New Documentation For Mystery Crawler

Google updated their documentation with new information about a mysterious crawler used by Google that was not widely known until today

Google Safety Crawler

Google updated the list of their official crawlers by adding the name and information for a relatively unknown crawler that publishers have been seeing now and then but no documentation for it previously existed.

Although Google added official documentation for this crawler the information provided seems to encourage more clarification.

Special Case Crawlers

Google has several kinds of crawlers (also known as bots and spiders).

The different forms of crawlers are:

  • Common Crawlers
    These bots are mostly used for indexing different kinds of content. But some common crawlers are also for search testing tools, internal Google product team use, and crawling related to AI.
  • User-triggered fetchers
    These are bots that are triggered by users. This includes uses such as for fetching feeds or site verification.
  • Special-case crawlers
    These are for special cases like for mobile ads webpage quality checks or for push notification messages via Google APIs. These bots do not obey the global user agent directives in robots.txt that are signaled with the asterisk (*).

The new crawler documentation is for the Google-Safety user agent. The crawler is not new but the documentation is new.

Google-Safety Crawler

The documentation in the Special-case Crawlers Google-Safety crawler is one that’s used by Google’s processes for finding malware.

Unique among Special-case crawlers, the Google-Safety Crawler completely ignores all robots.txt directives.

The new documentation for the Google-Safety Crawler:

“The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties.

This user agent ignores robots.txt rules.”

The full agent string for the crawler:


Read the new documentation for the Google-Safety user agent on the Google Search Central page for crawlers in the section devoted to Special-case crawlers.

Overview of Google crawlers and fetchers (user agents) – Special-case crawlers



Category News SEO
SEJ STAFF Roger Montti Owner - at

I have 25 years hands-on experience in SEO and have kept on  top of the evolution of search every step ...

Google Adds New Documentation For Mystery Crawler

Subscribe To Our Newsletter.

Conquer your day with daily search marketing news.