Xenu is a highly valuable tool (that’s another great exception from the rule: it is both free and awesome). Let’s see how we can use it for SEO purposes:
1. Find broken links (to make your site easier to crawl and thus increase your crawl rate):
2. Identify your site internal duplicate content issues (duplicate titles):
3. Find largest pages/images (to optimize your page load time):
4. Find least linked-in pages (quick tip: to learn which exactly pages link to the current page, right-click / ALT+ENTER on the URL and choose “Properties”):
5. Find pages with most outlinks (increase your site crawl rate by minimizing the number of each page outbound links):
6. Find your deepest-buried pages and learn how easily they can be found (by “status” and “level” properties):
7. Find images that lack alternative text:
More tips:
- Right-click on an URL to get instant access to Google Cache and Internet Archive;
- CTRL+R will retry all failed URLs;
- Set less parallel threads in “Options”=> “Properties” to have fewer failed connections.
Comments
30 responses so far ↓
g1smd on Oct 20, 2008 at 8:03 am
I have been using Xenu for many years. It is invaluable. There’s yet more stuff you can do:
– Run a list of test URLs (saved in a text file) covering every possible URL *format* that might ever be requested (Note “every URL format” *not* “every URL”), to make sure that everything non-canonical gets bounced with either a 301 or 404 response. It is amazing how many ways a CMS might have to still generate a templated page full of random content, or even generate no content on a page with full outgoing navigation.
– Scan the site as normal, and look at the final report (generated to an HTML page) and then identify navigation links within the site that are hitting a redirect because they point to a non-canonical URL which already has a redirect in place to correct it.
– Look at the HTML report page, at the list of outgoing links to other sites, and identify broken 404 links to fix or delete, and outgoing links that hit a “stupid” redirect on the other site; one that can be easily corrected by changing the outgoing link URL here.
– Find errors where relative internal navigation points to the wrong place and can cause crawl errors.
– Find links within your site that you didn’t know existed. These occur in some design programs, where when you delete a link, only the anchor text is deleted. In the HTML code the link still remains like ["a href="somepage.html"][/a] and is still pointing at some URL (which may or may not exist) but which search engines may treat as being a “hidden” link.
– Find bot traps, with infinite content, such as an unbounded calendaring system. I hit one of those recently; managed to spider the calendar back to the 1700s and forward to the 2300s with every page (except for those in 2007 and 2008, and early 2009 ) being devoid of any unique content.
– Find images that are waaay too big. I have caught someone with 1 MB files in the “thumbs” folder before now.
– Check the operation of bot traps, where those limit speed of spidering, or number of pages accessed (there are better tools for that, but this can be useful sometimes).
– Generate a basic HTML sitemap (look at the HTML report). If the visual structure of the sitemap isn’t close to that you had envisioned for your site, you have problems with your internal linking. Especially look for items in the Sitemap that say “301 Moved” or “302 Moved” too.
Yes. A very useful tool and one that I use nearly ever day for something or other.
SEO Mumbai on Oct 20, 2008 at 8:28 am
Thanks Ann and g1smd for your tips. Xenu is really wonderful tool which can help us for both on-page as well as off-page SEO
Jack on Oct 20, 2008 at 8:49 am
I’m always looking for good tools to help me with my “best practices” game. Half of SEO is making sure the onpage factors that we can control are all squared away. If this tool helps me make better pages I’ll use it a lot. Thanks for the tip
Amit Nyamtabad on Oct 20, 2008 at 10:18 am
Xenu is really invaluable for SEO research. I really liked point #4 and #6 a lot :)
Thanks a ton for sharing.
WebSite Design Orange County on Oct 20, 2008 at 10:26 am
Hmmmmmm, no Mac version. WTF! Looks like we’ll stick with ALM.
SEO company on Oct 20, 2008 at 10:33 am
Very interesting post.
Mher on Oct 20, 2008 at 12:10 pm
Has anyone used this tool for joomla based sites. I have had nothing but issues with this tool.. it always seems to crash.. is it because of the dynamic urls?
Thanks
g1smd on Oct 20, 2008 at 12:44 pm
I have never had Xenu LinkSleuth crash outright on me.
However, if there is a situation where a site has very bad duplicate content issues often caused by internal linking that results in an infinite loop of crawling, then the process that generates the HTML report can take many *days* to do its stuff - and may well run out of memory before that.
g1smd on Oct 20, 2008 at 12:49 pm
For Joomla-based sites, is the link checker spidering a part of your site that happens to be an infinite, unbounded, calendar? That can cause issues to any bot, but Googlebot et. al will abort after some while.
Sebastian on Oct 20, 2008 at 2:17 pm
Is it possible to teach it going on subdomains?
Arnie K on Oct 20, 2008 at 11:05 pm
I would be curious to know about search subdomains too. Great tool.
Bfri on Oct 21, 2008 at 1:51 am
I have come across the tool several several times but never had the time to really check it out. This is definitely a very useful post and provides me with another reason to give the tool a try. :) Thanks!
Also good sources to compare 404 reports with the reports from Google Webmaster Tools.
Software Testing on Oct 21, 2008 at 2:28 am
I recently used the same tool to find the number of pages in my website :)
Ben McKay on Oct 21, 2008 at 5:13 am
Hi Ann,
A great post. I use Xenu like many SEO’s loads but didn’t know it could do so much.
I love how the uglier the tools the better they perform too…it makes an interesting graph.
Remember how ugly Firefox used to be?!
Ben M
Jeff on Oct 21, 2008 at 6:34 am
Ann,
Thanks for letting us know about this tool. I installed it and within a few minutes I was able to fix some problems with my web site that I was unaware of. Thanks again SMARTY! :)
Jeff
SeekGeek on Oct 21, 2008 at 7:48 am
Great Post Ann
What would be really awesome is if it could show only “follow” outlinks (point 5) i.e. outlinks excluding “no follows”
SeekGeek on Oct 21, 2008 at 8:39 am
The other thing that would improve it plenty is the ability to sort by more than 1 column (like excel’s sort by…and then by.. function) So you could sort by eg. type then title.
Oh well, its a freebie, can’t be too demanding…
Matt Inertia on Oct 21, 2008 at 8:51 am
Been using xenu religiously on sites for the last year. Great article.
Tamil Movie News Online on Oct 21, 2008 at 10:44 am
Interesting post and valuable information about Xenu tool. Thanks Ann
Rathis on Oct 21, 2008 at 10:47 am
Great tips! Very helpful for me. Thank you very much.
cw360 on Oct 21, 2008 at 6:06 pm
Xenu in invaluable to me. I have been using it for about a year now. I would not be able to maintain the sites I maintain without it.
Imathon on Oct 22, 2008 at 9:05 am
Totally agree with SeekGeek. Excluding “nofollow” is a must, hopefully the next feature!
g1smd on Oct 24, 2008 at 4:29 am
*** What would be really awesome is if it could show only “follow” outlinks (point 5) i.e. outlinks excluding “no follows” ***
*** Totally agree with SeekGeek. Excluding “nofollow” is a must, hopefully the next feature! ***
You are totally missing the point. This tool is for checking that the links actually *work*, and that your aren’t linking to a URL that returns 404 or a redirect. So, every link should be followed so that it can report which ones are useless to the people who click on them.
SeekGeek on Oct 24, 2008 at 5:00 am
@g1smd the title of ann’s article was “7 things you can do with xenu…” not ” 1 thing…” Yes, locating 404s is the main thrust of the prog (as covered in point 1) but when counting outbound links (point 5) the “nofollow” variable is critical info.
Kevin Kirkpatrick on Nov 5, 2008 at 5:58 pm
I’m trying to figure out how to have Xenu limit the pages it crawls. For example, if I just want it to crawl the homepage and then stop. Then from there I could say ok, now go crawl these pages. Instead of doing it backwards by limiting every page.
Neil on Nov 26, 2008 at 4:47 pm
Great post. Thanks AS.
nuevojefe on Dec 28, 2008 at 6:03 pm
If crawling very large sites I would recommend going to Options >> Preferences >> Advanced Tab and unchecking “Fail all URLs with Failed Host” so you can leave it running overnight (for example) and not come back to find that it finished prematurely because of a temporary response problem.
It can depend on the machine obviously. I had a few year old cpu running xenu for me and when the URL count got around 800,000 it would tend to crash. I just recommend pausing and saving the file every now and then for large projects.
Ahmed on Feb 6, 2009 at 5:10 am
I just finished exploring XENU 1.3 and found it really helpful in testing web applications. We also tried Xenu for Site menus (such as drop downs ,site navigation menus etc) but were unable to get any result. Can you please confirm if we could infact use it for site navigation menu testing as well?(without manually clicking on each menu sub level individually)The site menus are written in JavaScript while each sub level menu points to some live urls (which are obviously hard coded)o I wanted to inquire is there a way or any Xenu feature , through which Fiddler can check all urls/link specified in Menu drop down without manually clicking them
Mark on May 14, 2009 at 6:54 am
Thank you for this article. I discovered Xenu a couple of days ago and it has taken me that long to correct my broken links to external sites which I didn’t know I had - hopefully my site will improve as a result. I read your article to learn all the features so I get the most out of it.
Andrew on May 26, 2009 at 12:25 am
I’ve had Xenu installed for a while now but I hadn’t actually used it much until today. As usual Ann, your tips are extremely helpful (and g1smd) and have made my task of analysing our link profile a whole lot easier. Thanks!
Leave a Comment