Is Your Website Being Indexed Properly by Google?

SMS Text
google-logo

One of the most common problems experienced when trying to rank in Google, is that your website is not currently being indexed correctly. If this is the case, it means Google is failing to access your web pages to index your site’s content effectively.

To check whether your site is efficiently crawled and listed, you will need to log into your Google Webmaster Tools and check the “Google index” tab. There you will find the total number of pages the search engine has indexed. If you see a drop in the number of these pages, you are likely to experience a decrease in traffic levels.

google homepage

Finding the Reason Behind Your Indexing Issues

If you’ve taken a look at your Webmaster Tools and it’s clear that not all your pages are being found by Google’s crawlers, now is the time to take a closer examination at the possible problems Google is experiencing with your website.

Does Your Site Have Crawler Errors?

To find out if Google is indexing your site fully, begin by heading to your Google Webmaster Tools dashboard and checking your Crawler Error messages. The most-likely error message you will find is a 404 HTTP Status Code warning. It signals that the URL cannot be found.

Other crawling errors include:

  • Robots.txt – A   poorly scripted Robots.txt file can be detrimental to your Google indexing. This text file is like a set of instructions telling a search engine crawler not to index parts of your website. If it includes a line such as “User-agent: *Disallow: /” this basically tells every single crawler it experiences to ‘get lost’ – including Google.
  • .htaccess – This invisible file can do nasty things if incorrectly configured on your site. Most FTP clients allow you to toggle hidden/seen files so that you can access it if required.
  • Meta Tags – If you have pages that aren’t being indexed, be sure they don’t have the following meta tags in the source code: <META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
  • Sitemaps – If you receive a Sitemaps crawling error, it means your website sitemap is not updating properly; your old sitemap is being repeatedly sent to Google instead. When you’ve tackled any issues signalled by the Webmaster Tools, make sure you run a fresh sitemap and re-submit it.
  • URL Parameters – Google allows the option to set URL parameters when it comes to dynamic links. However, incorrect configuration of these can result in pages that you do want picked up being dropped instead.
  • DNS or Connectivity issues – If Google’s spiders simply can’t reach your server, then you may encounter a crawler error. This could be for a variety of reasons such as your host is down for maintenance or had a glitch of their own.
  • Inherited Issues – If you have bought an old domain or moved your website to an old website’s location it is possible the previous site had a Google penalty. This will inhibit indexing of the new site. You will have to file a reconsideration request with Google.

If you are considering using a historic domain for your site, be sure to take a look at its history before purchasing. You can make use of the Internet Archive’s Wayback Machine to see pages that were previously hosted on your domain.

Does Your Site Have Syntax Errors or Structural Complications?

Google is very tolerant when it comes to HTML mark-up mistakes within webpages, but it is possible that syntax errors can prevent indexing (in extreme cases). Check your site’s HTML with the W3C’s HTML Validator to see a report of errors you need to correct.

Google advises you make your site structure as logical as possible. Every page should be reachable from at least one test link. You can use a text browser, like Lynx, to look at your site much the same way the spiders see it. Remember, the parts of your site that use frames, JavaScript, Flash, session IDs, cookies and DHTML may be missed by crawlers.

Does Your Site Have Inbound Links?

To be indexed with Google, your website needs to have at least one quality inbound link from another website already indexed in the search engine. This is a common reason it takes a lot of new websites a while to be successfully indexed.

One way to create some quick links is to update social networks with your website URL or add a link on an existing related website that you own. Social media profiles that carry high weight include: Facebook pages, Twitter profiles, Google+ profiles/pages, LinkedIn profiles, YouTube channels, and Pinterest profiles.

Offsite content is another excellent way to build links that will help your site get indexed properly. Offsite content is content relevant to your site that is hosted elsewhere: guest posts on other blogs in your niche. Just keep in mind, you need to make sure these external sites are all high quality, as links from ‘spammy’ sites will do your website harm instead of good. The best way to ensure your links are high quality is to make sure that they have ‘natural links’, links that develop as part of the dynamic nature of the internet where other sites link to content they find valuable.

See Google’s Webmaster Guidelines for a more in-depth understanding of what they consider to these to be.

Has Google Penalized You?

One of the most difficult obstacles in proper indexation by Google is a Google penalty. There are a number of reasons why you might encounter a penalty from Google, but if you do not deal with the issue they raise, you may be deindexed (removed from their search engines).

Avoid Google Penalties by Steering Clear of The Following Techniques:

  • Automatically generating content
  • Link schemes
  • Plagiarizing or duplicating content
  • Cloaking
  • Sneaky redirects
  • Hidden links & text
  • Doorway pages
  • Content scraping
  • Affiliate programs with little content value
  • Using irrelevant keywords
  • Pages that install trojans, viruses, & other adware
  • Abusing rich snippets
  • Automating queries to Google

Recovering from Google penalties requires hard work and due diligence to remove links on your part; you will need to submit a reconsideration request before your site is effectively indexed and ranked once more.

Fix Your Indexing

Most of these checks are quick and easy to make, so don’t let your SEO and link building efforts go to waste – make sure your website is indexed correctly by Google. It’s surprising how many websites make some of the smallest mistakes, and it prevents their site from being indexed correctly. In the end, it hurts their website’s rankings, which hurts their traffic, which hurts their sales.

Image Credits

Featured Image: Google logo via Wikipedia
Image #1: Google homepage via Wikipedia

Barrie Smith
Barrie Smith is a Link Building guru at Receptional Ltd and has over 11 years of experience in Digital Marketing. Barrie has worked on several... Read Full Bio
Barrie Smith
Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • Useful, thorough info. Thank you! 🙂

  • Great post Barrie! I can’t believe how many clients I see that don’t realize their websites are not being indexed properly. In fact just this weekend I wrote an article on how to fix your images indexing on WP Engine when you have the CDN enabled.

    I’m pretty sure most people running on WP Engine haven’t even bothered to fix this… which is a problem for them lol. They are missing out on a lot of extra traffic.

  • Excellent post Barrie. I am in the process of submitting my sitemap to Google Webmaster Tools. I have a question: would you submit your site as http://www.yourwebsite.com/sitemap.xml or http://www.yourwebsite.com/sitemap_index.xml

    Is there a preference as to which indexes all the sub pages, categories, tags, images etc?

    Thanks.

    Kevin

    • Kevin:
      Either sitemap will work fine. Make sure you check the sitemap for any errors once Google webmaster tools updates your new sitemap. I would recommend checking you webmaster tools each month.

      • You can include: sub pages, categories, tags, images etc. I usually only use pages, post, images, videos.

    • Barrie Smith

      Hi Kevin,

      Similar to Jonathan has said before me, either sitemap is fine.

      As far as I know, the file name isn’t important, only that the sitemap is configured properly.

      Regards,

      Barrie

    • Adding your RSS feed as a sitemap will help as well.

  • Great post, Barrie.
    Diving into the Google index we can notice that the big G crawls and indexes the web pages,with the help ,of backlinks. As initialization, it crawls some authoritative sites and then crawl, index all the links in that page.
    In the past, there was to get indexed in Google without inbound links, however, now blog or site without links gets indexed. How’s this possible?

    Many of the webmasters don’t every monitor Webmaster tools,or check Google index. If it continues for a long time it would be a problem.

    • Barrie Smith

      Hi Akshay,

      Your website can still be indexed in Google without links. For instance, submitting a sitemap via Webmaster Tools might result in indexation with no links. Likewise using the “fetch as Google” tool in WMT. It’s possible that registering an Analytics property could add your URLs to Google’s crawl list. Data tracked from Chrome and those logged into Google might also be used to generate lists of new URLs to crawl. For the official explanation though, see: https://support.google.com/webmasters/answer/70897

      Either way, you’re unlikely to rank well and get traffic in Google if you have not built a website that is inheriting links. If it has 0 links after a few weeks then as far as I am concerned, no one is interested or a very poor marketing job has been done.

      Regards,

      Barrie

  • Barrie, excellent post!
    I wonder what’s your opinion on these services that scan your blog feed and post links to your google+, twitter, facebook, etc.? Does that help and are those domains going to outrank the original potentially?

    • Barrie Smith

      Hi Savas,

      Those domains could possibly outrank the original, but it is an unlikely scenario, so long as the original is well-optimised.

      Regards,

      Barrie

  • Urban Media

    Nice post Barrie. It can often be the case that you need to build your website with this in mind. Without building your website for SEO, it can be likely that Google will not be able to properly rank your website. Ensure you have submitted site maps, Google Webmaster Tools is a real help to anyone who has an existing website. Inbound links from good quality directories will definitely help also. Good work!

  • Floyd Allen

    For easy check up: site:example.com
    Thanks for the post Barrie 🙂

  • hello, i have a problems with my some urls. after modified some urls i always get 404 urls did not found. How i solve this issue?

  • Great post Barrie.!!! Thanks for useful information. made it very simple and clear for people like me.

  • Hey Richard,
    Any tips for getting Google to index pages on other websites that link to you?
    I’ve recently linked to these pages from G+ pages which has encouraged Google to crawl them, it’s worked but it’s not an ideal method..

  • This issue here is how google change the algoritmo, and a small mistake cab be very bad for our site,,also google make seo expert confuse with change on the tactics..

  • Prashant Aier

    Thanks for the post. I have one question. I have added some new pages to my site and with using “fetch as Google” it started to show in Google search results with specific key words. But after a week, the pages are not showing under that specific key words. When i tried Specific key words + “mywebsitename”, Google search results shows the pages. It means my pages are indexed but they are yet to rank. If i am correct, how much time normally takes to rank the pages ?