To ensure that search engines understand your website structure correctly you should control each page HTTP headers. Your server should handle all requests properly – by this you will both achieve better crawl rate and higher rankings.
Understanding HTTP header status codes:
I don’t want to go into too much details here and describe only what is really important for SEOs. Basically there are 5 main status code classes:
- informational (1xx);
- successful (2xx) request;
- redirection (3xx) = further action needs to be taken by the user agent in order to fulfill the request;
- client error (4xx);
- internal server error (5xx).
Out of the above status codes SEOs most often have to deal with the following ones:
- 200 = the request has succeeded;
- 301 = moved permanently;
- 404 = not found.
It is necessary to make sure that in each particular case the correct status code is returned: e.g. don’t use redirect for non-existent pages – return an optimized page with 404 header code.
Cases SEOs Should Look out for:
Most often the following cases may result in 200 status code when handled improperly:
- index.html vs. directory root;
- www vs non-www version;
- double slashes in the file path;
- mixed up letter cases;
- non-existent pages;
- https protocol requests; etc.
HTTP Header Checking Tools:
There are numerous web-based tools that will show you HTTP header for any URL, a few examples include:
There is also one handy FireFox extension that allows to view all http requests issued by the Web page loaded in my browser: it adds information “Headers” tab under “Page Info” of a web page.
Ann,
Simple but effective write up! The Firefox extension comes in handy. Thanks for sharing.
@Ann,
would suggest 302 to the list that SEOs must often deal it shouldn’t be there. Also how abt customized 404 page?
I think 307 and 410 also deserve a mention these days, and I’ll second the call for mentioning 302 and why it can be a bad thing to use.
@Raghavan, @g1smd : agreed, I also gave the link to the full list of status codes in the post and here it is also.
Other good tools to check headers:
Charles (OS X): http://www.charlesproxy.com/
does some sweet stuff, acts as a proxy between the web and client capturing http requests
Wireshark: http://www.wireshark.org/
goes way deeper and acts as a complete network diagnostics tool
And of course my favorite of all, good old wget -S
nice fireox add-on, it is often useful to know more about http header when debugging websites
Good job, Ann. The information about title and keywords was most useful. I was surprised that 3- and 4 -word keyphrases were considered and didn’t like the results my page produced. Back to the drawing board!
Thanks for your insight.
Frederick