SEO · Tools

7 Things You Can Do With Xenu Link Sleuth

Xenu is a highly valuable tool (that’s another great exception from the rule: it is both free and awesome). Let’s see how we can use it for SEO purposes:

1. Find broken links (to make your site easier to crawl and thus increase your crawl rate):

xenu broken links1 7 Things You Can Do With Xenu Link Sleuth

2. Identify your site internal duplicate content issues (duplicate titles):

xenu 7 Things You Can Do With Xenu Link Sleuth


3. Find largest pages/images (to optimize your page load time):

xenu size 7 Things You Can Do With Xenu Link Sleuth

4. Find least linked-in pages (quick tip: to learn which exactly pages link to the current page, right-click / ALT+ENTER on the URL and choose “Properties”):

xenu inlinks 7 Things You Can Do With Xenu Link Sleuth

5. Find pages with most outlinks (increase your site crawl rate by minimizing the number of each page outbound links):

xenu outlinks 7 Things You Can Do With Xenu Link Sleuth

6. Find your deepest-buried pages and learn how easily they can be found (by “status” and “level” properties):

xenu deep pages 7 Things You Can Do With Xenu Link Sleuth

7. Find images that lack alternative text:

xenu missing alt text 7 Things You Can Do With Xenu Link Sleuth

More tips:

  • Right-click on an URL to get instant access to Google Cache and Internet Archive;
  • CTRL+R will retry all failed URLs;
  • Set less parallel threads in “Options”=> “Properties” to have fewer failed connections.
f8d69258525dec38624a29eb3d570d8c 64 7 Things You Can Do With Xenu Link Sleuth
Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as a base for her writing, tutorials and her guest blogging project, MyBlogGuest.com.
f8d69258525dec38624a29eb3d570d8c 64 7 Things You Can Do With Xenu Link Sleuth

You Might Also Like

Comments are closed.

41 thoughts on “7 Things You Can Do With Xenu Link Sleuth

  1. I have been using Xenu for many years. It is invaluable. There’s yet more stuff you can do:

    – Run a list of test URLs (saved in a text file) covering every possible URL *format* that might ever be requested (Note “every URL format” *not* “every URL”), to make sure that everything non-canonical gets bounced with either a 301 or 404 response. It is amazing how many ways a CMS might have to still generate a templated page full of random content, or even generate no content on a page with full outgoing navigation.

    – Scan the site as normal, and look at the final report (generated to an HTML page) and then identify navigation links within the site that are hitting a redirect because they point to a non-canonical URL which already has a redirect in place to correct it.

    – Look at the HTML report page, at the list of outgoing links to other sites, and identify broken 404 links to fix or delete, and outgoing links that hit a “stupid” redirect on the other site; one that can be easily corrected by changing the outgoing link URL here.

    – Find errors where relative internal navigation points to the wrong place and can cause crawl errors.

    – Find links within your site that you didn’t know existed. These occur in some design programs, where when you delete a link, only the anchor text is deleted. In the HTML code the link still remains like ["a href="somepage.html"][/a] and is still pointing at some URL (which may or may not exist) but which search engines may treat as being a “hidden” link.

    – Find bot traps, with infinite content, such as an unbounded calendaring system. I hit one of those recently; managed to spider the calendar back to the 1700s and forward to the 2300s with every page (except for those in 2007 and 2008, and early 2009 ) being devoid of any unique content.

    – Find images that are waaay too big. I have caught someone with 1 MB files in the “thumbs” folder before now.

    – Check the operation of bot traps, where those limit speed of spidering, or number of pages accessed (there are better tools for that, but this can be useful sometimes).

    – Generate a basic HTML sitemap (look at the HTML report). If the visual structure of the sitemap isn’t close to that you had envisioned for your site, you have problems with your internal linking. Especially look for items in the Sitemap that say “301 Moved” or “302 Moved” too.

    Yes. A very useful tool and one that I use nearly ever day for something or other.

  2. I’m always looking for good tools to help me with my “best practices” game. Half of SEO is making sure the onpage factors that we can control are all squared away. If this tool helps me make better pages I’ll use it a lot. Thanks for the tip

  3. Has anyone used this tool for joomla based sites. I have had nothing but issues with this tool.. it always seems to crash.. is it because of the dynamic urls?

    Thanks

  4. I have never had Xenu LinkSleuth crash outright on me.

    However, if there is a situation where a site has very bad duplicate content issues often caused by internal linking that results in an infinite loop of crawling, then the process that generates the HTML report can take many *days* to do its stuff – and may well run out of memory before that.

  5. For Joomla-based sites, is the link checker spidering a part of your site that happens to be an infinite, unbounded, calendar? That can cause issues to any bot, but Googlebot et. al will abort after some while.

  6. I have come across the tool several several times but never had the time to really check it out. This is definitely a very useful post and provides me with another reason to give the tool a try. :) Thanks!
    Also good sources to compare 404 reports with the reports from Google Webmaster Tools.

  7. Hi Ann,

    A great post. I use Xenu like many SEO’s loads but didn’t know it could do so much.

    I love how the uglier the tools the better they perform too…it makes an interesting graph.

    Remember how ugly Firefox used to be?!

    Ben M

  8. Ann,
    Thanks for letting us know about this tool. I installed it and within a few minutes I was able to fix some problems with my web site that I was unaware of. Thanks again SMARTY! :)
    Jeff

  9. Great Post Ann

    What would be really awesome is if it could show only “follow” outlinks (point 5) i.e. outlinks excluding “no follows”

  10. The other thing that would improve it plenty is the ability to sort by more than 1 column (like excel’s sort by…and then by.. function) So you could sort by eg. type then title.

    Oh well, its a freebie, can’t be too demanding…

  11. Xenu in invaluable to me. I have been using it for about a year now. I would not be able to maintain the sites I maintain without it.

  12. *** What would be really awesome is if it could show only “follow” outlinks (point 5) i.e. outlinks excluding “no follows” ***

    *** Totally agree with SeekGeek. Excluding “nofollow” is a must, hopefully the next feature! ***

    You are totally missing the point. This tool is for checking that the links actually *work*, and that your aren’t linking to a URL that returns 404 or a redirect. So, every link should be followed so that it can report which ones are useless to the people who click on them.

  13. @g1smd the title of ann’s article was “7 things you can do with xenu…” not ” 1 thing…” Yes, locating 404s is the main thrust of the prog (as covered in point 1) but when counting outbound links (point 5) the “nofollow” variable is critical info.

  14. I’m trying to figure out how to have Xenu limit the pages it crawls. For example, if I just want it to crawl the homepage and then stop. Then from there I could say ok, now go crawl these pages. Instead of doing it backwards by limiting every page.

  15. If crawling very large sites I would recommend going to Options >> Preferences >> Advanced Tab and unchecking “Fail all URLs with Failed Host” so you can leave it running overnight (for example) and not come back to find that it finished prematurely because of a temporary response problem.

    I have never had Xenu LinkSleuth crash outright on me.

    It can depend on the machine obviously. I had a few year old cpu running xenu for me and when the URL count got around 800,000 it would tend to crash. I just recommend pausing and saving the file every now and then for large projects.

  16. I just finished exploring XENU 1.3 and found it really helpful in testing web applications. We also tried Xenu for Site menus (such as drop downs ,site navigation menus etc) but were unable to get any result. Can you please confirm if we could infact use it for site navigation menu testing as well?(without manually clicking on each menu sub level individually)The site menus are written in JavaScript while each sub level menu points to some live urls (which are obviously hard coded)o I wanted to inquire is there a way or any Xenu feature , through which Fiddler can check all urls/link specified in Menu drop down without manually clicking them

  17. Thank you for this article. I discovered Xenu a couple of days ago and it has taken me that long to correct my broken links to external sites which I didn’t know I had – hopefully my site will improve as a result. I read your article to learn all the features so I get the most out of it.

  18. I’ve had Xenu installed for a while now but I hadn’t actually used it much until today. As usual Ann, your tips are extremely helpful (and g1smd) and have made my task of analysing our link profile a whole lot easier. Thanks!

  19. I have got some crazy problems where two of my posts are causing huge headaches. They are creating hundreds of links with the same category name over and over e.g. ../../featured/featured/featured/../../../post-name

    The worst thing is that every single combination of the links actually opens up the article.
    I’ve attached a screenshot.
    http://forsaleitems.s3.amazonaws.com/duplicate-content.gif

    Please help. I’m desperate.

  20. It’s quite a nice tool thanks Ann for showing us how many other functions it has just found it yesterday and i am still learning how to use it.

  21. Great post ann, and i advise people who are having issues with larger sites to trial the 64 bit version of the application, it seems to run much better if you have a 64 bit system.

  22. Amazing tool, I've been using it for a while but I don't know it is capable of so many different yet interesting things. Many thanks!

  23. thanks buddy,really xenu is very useful ,i hav been using xenu from last couple of month,but today i got 2 know all the fields abt xenu thanks buddy

  24. thanks buddy,really xenu is very useful ,i hav been using xenu from last couple of month,but today i got 2 know all the fields abt xenu thanks buddy

  25. Thanks for the wonderful article. Could you please mention how can we manage the Xenu, if account authentication is required.
    Thanks