How to Switch Your User-Agent to Googlebot

July 8th, 2008 by Ann Smarty | 14 Comments

Sometimes the issue with your site is right there but you can’t find it. You check code, backlinks and website architecture but can’t find the answer. What most webmasters fail to do is to try visiting the page with the user-agent being Googlebot.

By switching your user-agent to Googlebot, you will essentially see your site in the same eyes as Google, so if there is a Google specific flaw, some cloaking going on, or possibly even a Googlebot targeted hack, switching to user-agent Googlebot will help identify these issues.

1. User-agent switcher - a FireFox extension also available for Flock and Seamonkey that sends the server different information about which user-agent is requesting a file. It should be enhanced with the list of user-agent list.

User-agent switcher

2. Header detector that both allows you to change the user-agent and shows you the header response for the set user-agent:

Header detector

Use the above tools to:

find issues with your own site;
catch other people’s black-hat tactics, i.e. cloaking.

Just a couple of cases illustrating how these tools can help to solve the problem:

When Google Comes Up Number One for Your Domain Name - a puzzle solving why Google ranks itself first for [domain] search of a site. With the help of the user-agent switcher it was found that the site was cloaking for Googlebot.
What Google Doesn’t See CAN Hurt You - a post describing how Google dropped one perfectly legit subdirectory out of its index because it was cloaking to Google, but then sending Google to a 404 error page.

Vote for this post : 3

or Buzz it at Yahoo :

Comments

14 responses so far ↓

Aqeel Malik on Jul 8, 2008 at 8:01 am

Hello Ann,

Thanks for sharing some of google flaws, really its to have such information.

But sorry i could not understand the procedure to that how i can check it from Mozilla after installing that particular extension.

Please add a short detail of it.
Regards
Ann Smarty on Jul 8, 2008 at 9:29 am

@Aqeel : that’s really very easy. All you have to do is to choose the user-agent (Tools => User Agent Switcher) and load the page.
Tom Lynch on Jul 8, 2008 at 3:33 pm

Hi Anne,

Re: User-agent switcher - do you have any example sites that look different in different user agents. I’m in FF on a PC XP not seeing any difference anywhere…

Regards,

Tom
Halfdeck on Jul 8, 2008 at 10:15 pm

Thanks for the post Ann. I somehow missed those SEOmoz posts - alot of interesting stuff in both.

@Aqeel, if by “exposing Google flaws” you’re referring to those two SEOmoz posts, they expose flaws with the websites in question, not Google.
Aqeel Malik on Jul 9, 2008 at 2:28 am

@ Ann, Thanks for guiding me.

@ Halfdeck.
yea i was mentioning the same.. Sorry for any misconception.
Indian art and artist on Jul 9, 2008 at 2:41 am

good article
Bizinfotech on Jul 9, 2008 at 2:42 am

I had used user-agent in my website. thanks
Pierre Far on Jul 9, 2008 at 9:29 am

Except if the site actually authenticates any request that looks like GBot. Then you don’t know what you’re going to get.
Schmoozii on Jul 9, 2008 at 5:53 pm

Thanks for this post. I actually saw this on extension on another seo blog earlier and installed it. But then I didn’t know what it did so I uninstalled. Now I know. Thanks
jakob on Jul 10, 2008 at 3:27 am

Good post, but cant import the xml. Have tried to disable most of the FF (2.0.0.15) extensions and even reinstalled the User Switcher (0.6.11), but nothing. When I click import and choose the XML it simply goes back to the Option screen and does not import anything. Any ideas?
Aqeel Malik on Jul 10, 2008 at 7:06 am

The User Agent Switcher is present in Netscape latest version and the previous one also.
If you are having any problem while testing in Mozilla FireFox, try it in Netscape.
Ann Smarty on Jul 10, 2008 at 8:03 am

@jacob : have never heard of problems like that. Does your option screen looks this way?
Jakob on Jul 10, 2008 at 8:57 am

@ann: yes the option screen looks exactly like that. I click on “Import” choose the XML file click OK and it takes me back to that same Option screen without adding the additional agents…
fresh-warez.blogspot.com on Sep 11, 2008 at 8:52 am

Thank you for this tutorial