Start Free Trial
Advertisement
  1. SEJ
  2.  » 
  3. SEO

Can Being Crawled by Googlebot Desktop Cause Indexing Delay?

John Mueller was asked if getting crawled by Googlebot Desktop can explain why they're suffering from delayed indexing

Can Being Crawled by Googlebot Desktop Cause Indexing Delay?

Google’s John Mueller answered a question about whether it matters to indexing speed if Googlebot mobile or Googlebot desktop crawls a page. The person asking the question was worried about slow indexing and wanted to know if getting crawled by Googlebot desktop (instead of Googlebot mobile) was the reason.

Googlebot Desktop and Mobile

Google has a multitude of website crawlers that follow from link to link discovering new web pages to crawl and index.

If you look at your traffic logs you might see:

  • Googlebot Desktop
  • Googlebot Smartphone
  • Googlebot Image
  • Mediapartners-Google (Mobile AdSense)
  • FeedFetcher-Google
  • Google-Read-Aloud
  • AdsBot-Google
  • Googlebot-Video
Advertisement
Continue Reading Below

Those are just a few of the Google-related Google crawlers that can visit a website.

The question John Mueller answered was primarily concerned with Googlebot Desktop and Googlebot Smartphone and what impact on indexing the desktop version might have.

This is the question:

“Our website’s posts are being indexed after one week. The site is old and mobile-friendly.

When I submit URLs in Inspection Tools, it shows it’s crawled by Google smartphone.

But in the About section, the indexing crawler is shown as Googlebot Desktop.

Does this mean the Smartphone bot is crawling the URL and the Desktop bot is indexing this URL?

Is this the reason behind the indexing delay?”

Advertisement
Continue Reading Below

John Mueller’s answer:

“This would not be a reason for any kind of indexing issues.

I don’t know offhand why you might see this mix of desktop and mobile crawlers there.

It might just be that the Search Console is essentially saying well, by default, the Inspection Tool will just use the mobile Googlebot.

In practice, we always crawl with both of these crawlers.

It’s just a matter of how much we crawl and index with any of these crawlers.

So, usually you’ll see something like 80% is coming from mobile Googlebot and 20% maybe from desktop Googlebot.

And that kind of ratio, that is essentially normal and doesn’t mean that things are slower than anything else.”

Related: Why GoogleBot Doesn’t Crawl Enough Pages on Some Sites

Multiple Versions of Googlebot Crawl Websites

Google’s Googlebot support page states that both versions of Googlebot visit web pages.

According to Google’s official documentation:

“Your website will probably be crawled by both Googlebot Desktop and Googlebot Smartphone.

You can identify the subtype of Googlebot by looking at the user agent string in the request.”

According to Mueller and Google it’s totally normal to be crawled by both and there’s no indexing impact related to getting crawled by Googlebot Desktop.

Citation

Google’s Googlebot Page
Googlebot

Google’s Overview of Google Crawlers (User Agents)

Advertisement
Continue Reading Below

Does Predominantly Googlebot Desktop Crawl Cause Indexing Delay?

Watch the video at the 34:28 minute mark

CategoryNewsSEO
ADVERTISEMENT

Subscribe to SEJ

Get our daily newsletter from SEJ's Founder Loren Baker about the latest news in the industry!

Topic(s) of Interest*
By clicking the "SUBSCRIBE" button, I agree and accept the content agreement and privacy policy of Search Engine Journal.
Ebook

Roger Montti

Owner at Martinibuster.com

Roger Montti is a search marketer with over 20 years experience. I offer site audits, phone consultations and content and ... [Read full bio]

ADVERTISEMENT
Advertisement
Read the Next Article
Read the Next