To ensure that search engines understand your website structure correctly you should control each page HTTP headers. Your server should handle all requests properly – by this you will both achieve better crawl rate and higher rankings.
Understanding HTTP header status codes:
I don’t want to go into too much details here and describe only what is really important for SEOs. Basically there are 5 main status code classes:
- informational (1xx);
- successful (2xx) request;
- redirection (3xx) = further action needs to be taken by the user agent in order to fulfill the request;
- client error (4xx);
- internal server error (5xx).
Out of the above status codes SEOs most often have to deal with the following ones:
- 200 = the request has succeeded;
- 301 = moved permanently;
- 404 = not found.
It is necessary to make sure that in each particular case the correct status code is returned: e.g. don’t use redirect for non-existent pages – return an optimized page with 404 header code.
Cases SEOs Should Look out for:
Most often the following cases may result in 200 status code when handled improperly:
- index.html vs. directory root;
- www vs non-www version;
- double slashes in the file path;
- mixed up letter cases;
- non-existent pages;
- https protocol requests; etc.
HTTP Header Checking Tools:
There are numerous web-based tools that will show you HTTP header for any URL, a few examples include:
There is also one handy FireFox extension that allows to view all http requests issued by the Web page loaded in my browser: it adds information “Headers” tab under “Page Info” of a web page.