Sometimes the issue with your site is right there but you can’t find it. You check code, backlinks and website architecture but can’t find the answer. What most webmasters fail to do is to try visiting the page with the user-agent being Googlebot.
By switching your user-agent to Googlebot, you will essentially see your site in the same eyes as Google, so if there is a Google specific flaw, some cloaking going on, or possibly even a Googlebot targeted hack, switching to user-agent Googlebot will help identify these issues.
1. User-agent switcher – a FireFox extension also available for Flock and Seamonkey that sends the server different information about which user-agent is requesting a file. It should be enhanced with the list of user-agent list.
2. Header detector that both allows you to change the user-agent and shows you the header response for the set user-agent:
Use the above tools to:
- find issues with your own site;
- catch other people’s black-hat tactics, i.e. cloaking.
Just a couple of cases illustrating how these tools can help to solve the problem:
- When Google Comes Up Number One for Your Domain Name – a puzzle solving why Google ranks itself first for [domain] search of a site. With the help of the user-agent switcher it was found that the site was cloaking for Googlebot.
- What Google Doesn’t See CAN Hurt You – a post describing how Google dropped one perfectly legit subdirectory out of its index because it was cloaking to Google, but then sending Google to a 404 error page.