Migrating to a New Domain and URL Structure – My Learnings

Recently the company I worked for made a big decision to rebrand and with it came the dreaded domain name change. We are also having a new website being built so another SEO favourite URLs. This article will share my leanings from my sleepless nights and epic amount of research into making sure it didn’t go wrong.

After my research I decided to not listen to what I read and decided to change our domain name and page paths all in one hit. I bet you’re thinking “That’s suicide”, well yes but my results further on may surprise some.

Post Migration Preparation

I know of all too many bad experiences of this, from things like pages dropping forever to massive Google penalties for doing it wrong, which is why the prep work into this was huge. Just from my research I was getting worried about the task at hand. On our website we have over 70,000 URL’s. There were also a lot of rule sets for the different types of url paths our development team created the rules based on my rule sets and there was a huge amount of testing on a server that we had set up purely for testing.

Also being a solo In House SEO it can be hard to bounce ideas off people; luckily I have some good contacts through LinkedIn and Networking that I conversed with on a regular basis on these matters. I am also willing to do the same should you need the reassurance of what you are doing. (Details at the bottom of this post)

A big BIG URL List

Initially you will need a comprehensive URL list. You can get these either by spidering the site using tools such as Xenu or by asking your developers to create a script that will query your CMS’s database to generate a list of EVERY page on your website. But beware here, if your computer is of an average standard and you have a large site. Prepare for Excel to turn into a cripple when you need to work with the data.

With this list and the rules creates you can magic up some formulas that will create the new URL’s in the cell next to the old url. (This list will prove to be a bible for your PPC manager and affiliates if you have them) For me I used it for easy reference when people asked me “What’s the new URL for xyz page?”

If you have a smaller site then this will begin to form the basis of your 301 list. If you need to know how do 301’s then please Google it based on your server architecture. (As that’s another article separate to this) If in my case you have a large site then you need to sit down with your development teams and run through redirect rules based on your new URL construction rules. Prepare to call RegEx a friend by the time you finish this. (or just get your developers to do it if you are not sure)

Look at all those backlinks

Yet another long winded task in this project was updating of backlinks. In reality you cannot update all of your backlinks. If you have a portfolio of 100k backlinks in Yahoo you definitely have a lot of late nights. Luckily there are some awesome tools around to help you work out the importance of the backlinks. You can take your pick of SEOMoz’s Linkscape and Majestic SEO’s Link Intelligence as they are both brilliant tools, or like me, you can use both of the tools and mash them up in excel using some crazy formulas and pivot tables. From doing this I have worked out the top backlinks by order of importance. We’ll just store this for now until the day of the switch.

Testing

We have many development servers that we always test new functionality on. So with this task we dedicated on solely for the Migration Project. This server took a beating with the amount of spiders and load testing we threw at it. If you have the ability to do this then do, I cannot stress how much more weight off your shoulders it will be for you to see the success on there before putting it all live. What am I looking for in testing? Mainly I was looking to see any 404’s, 500’s that appear and to make sure the 301’s went through to the correct place. On a site as big as ours to check all results would have meant I’d be there for years. So we did a cross section of around 70% of our heavyweight pages and the rest being the lower converting pages.

Shane Jones
Shane Jones is an In House SEO working in for a leading Online Travel Agent in Manchester, England.
Shane Jones

Comments are closed.

15 thoughts on “Migrating to a New Domain and URL Structure – My Learnings

  1. Solid piece, Shane! The only thing that I would mention is that after doing dozens of these massive migrations, I’ve come to realize that the simple act of creating server-side redirects (301s in most cases) will ensure that the value of all historical inbound links will pass through to the new corresponding domain/urls.

    That realization helps save a ton of time and effort, because it means that you don’t have to go through the trouble of reaching out to the webmasters that linked to you in the past.

    Think of 301 as soldering or welding…it literally welds the old, deprecated URL to the corresponding new URL.

    Now, there is still value in reaching out to webmasters that have provided inbound links in the past, because a major site migration is a good excuse to get the anchor text of existing inbound links updated to include relevant keywords and phrases, but that’s a different story all together…

    1. 301 redirects don’t pass the full value of your inbound links, you lose some…best way is to contact webmasters and change the links!

  2. Totally agree, without the 301 this process would probably never happen. But luckily we do have it. This is probably the biggest thing I have done in my In House SEO career. Was really refreshing to pull it off successfully too

  3. Great post Shane. I did a similar migration for a news site and wanted to contribute one other point: if the site has had a sitemove in the past chances are there are existing redirects in place. I’d update these redirects to point to the new URL locations so you’re not going through multiple 301’s (i.e. page A – B, not A – B – C…).

    Also use MajesticSEO to look at old backlinks from previous migrations (order by date discovered) and make sure these pass through a single 301 to new URLs.

  4. I’d also make sure your error pages have your Analytics tracking code attached, it can help you identify missed redirects (in Google Analytics you can do this by looking at your content by title report – but make sure the title makes it clear its an error page) as well as prevent self referrels.

  5. Solid tips here Shane, and you can tell they’re borne from experience. I’m planning a similar migration to a client site now and planning ahead is certainly the name of the game. It is a lot more tricky when you’re not an in-house as you only have a small amount of time to look over problems and issues. Here are a few things I’d add on to your list that I’ve found useful:

    – Testing 301s on a test server – use Xenu to crawl all existing URLs on old site, export to a page map and open up in Excel. Then do a find and replace to your test server domain and then reimport all the URLs back into Xenu – this should give you an idea if you’ve missed any 301s that should be in place – without waiting for the site to go live before you notice

    – Set up a 404 notification system like http://www.linkpatch.com (free for a single domain) – by adding a bit of code to your 404 page this will ping you an email every time an actual user hits a 404 page and tell you where they came from – bloody useful at quickly picking up cracks that you missed out

    – One of the issues that comes from a domain / URL migration is waiting for search engines to actually pick up the redirects – for lower priority pages this can take a while for the 301 to be picked up and updated in the SERPs. One technique I like the idea of is <a href="http://www.thegooglecache.com/white-hat-seo/xml-sitemap-assisted-redirects-advanced-white-hat-seo/"XML Sitemap Assisted Redirects but recently WMT has been rejecting XML sitemaps containing redirected URLs – but I think you could create a temporary HTML sitemap with historical URLs so that these are recrawled within the new site. Be warned that Yahoo is rubbish at picking up 301s – I’ve seen old domains showing up literally years after 301s have been put in place.

    – Have patience… I’ve seen dropouts occur as a result of broad sweep changes such as these but if you’re doing this right the results will come back, and often better than before. But the worst thing you can do is panic if this temporary sandbox effect happens.

    Hope those are helpful =)

  6. @jaamit some good extras there, am setting up link patch now. Really really could be useful as we still have a handful of 404’s coming through.

  7. hi shane that was really nice post. Mate i need little help
    I have dedicated server and my 50 sites are hosted on same. Now due to some reason my server is showing in blacklisted category due to email spamming which in house guys did last year. When we send mail it goes to spam
    Only issue is Will it create negative impact in google rankings

    I mean will google penalize our site rankings because of email spamming
    My ip is 88.208.216.153 and i check on many sites like http://www.mxtoolbox.com
    please help

  8. Excellent article, actually a wall hanger for sure. All sound statements, that sadly most people never even think of when making a change.

    Thanks again, your blog is now bookmarked :)

  9. Hi Shane,

    Great article that will hopefully help alleviate some of the fears people have with this. It will always be a daunting task but your experience is definitely one I’ve encountered, thankfully.

    I don’t think you mentioned 301’s enough though…

    Just one thought that might be relevant: You did this change slap bang in the middle of the Google caffeine roll-out. Aside from you being braver than me in that respect (needs must), it may have had a bearing on the improved rankings. Have you had any -ve results since?

    Again, thanks for sharing.

    James.

  10. Awesome post!

    These projects are *very* time-consuming, and getting all of the details right before pushing the ‘deploy’ button is absolutely crucial.

    One very important point about the 301 redirects, is that any request for any of the old URLs, both canonical and non-canonical, must pass through a single 301 redirect to arrive at the final, correct, canonical new URL for that content.

    That is, canonicalisation rules for old URLs, those that are correcting non-canonical requests within the old site, also need to be modified to instead redirect to the new canonical URL on the new site. Any and all ‘redirection chains’ must be avoided where at all possible.

    That is (for example) the non-www to www redirect on the old site has to be modified to redirect non-www requests on the old site directly to www on the new site. Additionally, you might need either new rules for redirecting www on the old site to www on the new site or you will modify the existing non-www to www rule on the old site to cope with both non-www and www requests on the old site. Finally, there are the essential new rules for redirecting non-www on the new site to www on the new site.

    That’s a simple example. The complexity can become almost mind-boggling, and making lists of URLs and looking for patterns that can be redirected in bulk using a low number of rules is crucial. Rule ordering is also important, both in preventing conflicts and redirection chains, as well as increasing the efficiency of operation.

  11. Well done Shane glad it all turned out due to your hard work and thorough approach thanks for sharing your insights will us here. Ive added this to my blogroll. Cheers MauriceCFlynn

  12. Great Article Shane. My question is something you didn’t touch on. We purchased a domain form another company because it aligns with our branding. We set it up to redirect to our current domain about six months ago. I heard this was a bad idea as we need to build relevant history before migrating to the new domain. what are some thoughts on this? Should we ignore that and make the switch when we are ready?