SEO

Migrating to a New Domain and URL Structure – My Learnings

Recently the company I worked for made a big decision to rebrand and with it came the dreaded domain name change. We are also having a new website being built so another SEO favourite URLs. This article will share my leanings from my sleepless nights and epic amount of research into making sure it didn’t go wrong.

After my research I decided to not listen to what I read and decided to change our domain name and page paths all in one hit. I bet you’re thinking “That’s suicide”, well yes but my results further on may surprise some.

Post Migration Preparation

I know of all too many bad experiences of this, from things like pages dropping forever to massive Google penalties for doing it wrong, which is why the prep work into this was huge. Just from my research I was getting worried about the task at hand. On our website we have over 70,000 URL’s. There were also a lot of rule sets for the different types of url paths our development team created the rules based on my rule sets and there was a huge amount of testing on a server that we had set up purely for testing.

Also being a solo In House SEO it can be hard to bounce ideas off people; luckily I have some good contacts through LinkedIn and Networking that I conversed with on a regular basis on these matters. I am also willing to do the same should you need the reassurance of what you are doing. (Details at the bottom of this post)

A big BIG URL List

Initially you will need a comprehensive URL list. You can get these either by spidering the site using tools such as Xenu or by asking your developers to create a script that will query your CMS’s database to generate a list of EVERY page on your website. But beware here, if your computer is of an average standard and you have a large site. Prepare for Excel to turn into a cripple when you need to work with the data.

With this list and the rules creates you can magic up some formulas that will create the new URL’s in the cell next to the old url. (This list will prove to be a bible for your PPC manager and affiliates if you have them) For me I used it for easy reference when people asked me “What’s the new URL for xyz page?”

If you have a smaller site then this will begin to form the basis of your 301 list. If you need to know how do 301′s then please Google it based on your server architecture. (As that’s another article separate to this) If in my case you have a large site then you need to sit down with your development teams and run through redirect rules based on your new URL construction rules. Prepare to call RegEx a friend by the time you finish this. (or just get your developers to do it if you are not sure)

Look at all those backlinks

Yet another long winded task in this project was updating of backlinks. In reality you cannot update all of your backlinks. If you have a portfolio of 100k backlinks in Yahoo you definitely have a lot of late nights. Luckily there are some awesome tools around to help you work out the importance of the backlinks. You can take your pick of SEOMoz’s Linkscape and Majestic SEO’s Link Intelligence as they are both brilliant tools, or like me, you can use both of the tools and mash them up in excel using some crazy formulas and pivot tables. From doing this I have worked out the top backlinks by order of importance. We’ll just store this for now until the day of the switch.

Testing

We have many development servers that we always test new functionality on. So with this task we dedicated on solely for the Migration Project. This server took a beating with the amount of spiders and load testing we threw at it. If you have the ability to do this then do, I cannot stress how much more weight off your shoulders it will be for you to see the success on there before putting it all live. What am I looking for in testing? Mainly I was looking to see any 404′s, 500′s that appear and to make sure the 301′s went through to the correct place. On a site as big as ours to check all results would have meant I’d be there for years. So we did a cross section of around 70% of our heavyweight pages and the rest being the lower converting pages.

Ranking reports

Make sure you have set up your rank monitoring programs to run every day for a few weeks after you make the switch. This will help to identify anything instantly should it go wrong.

Webmaster Tools

Make sure you have your Webmaster accounts for the new domain set up and verified. Google especially not only because its the big one when it comes to Search Engines but also because they have the Change of Address {link to http://www.google.com/support/webmasters/bin/answer.py?answer=83106} function in there

Oh and Just in case…

Make sure your 404 page is a helpful one. Chances are some people will get some 404′s, in fact its highly likely with this risky a process. Make sure that your 404 page is a helpful one with links to some top level pages your link to sitemap and maybe even a site search.

Day of the Switch

So you’ve had your sleepless night and are probably up at a silly hour so as not to affect traffic. You get in and you deploy. The developers make the switch to the websites URL’s and Paths. It goes all tense until someone says shouts done. Now what…

Spider the site

First thing I did was browse the website, see if the page addresses had changed. I then went to Google and did a site:olddomain.com and clicked some of the top links. Using Firebug for Firefox I could see they were 301ing to their new homes.

To ensure it all further I now ran a web spider and waited. I used A1 sitemap generator to spider the site. When it was complete I had some results and also the ability to generate a sitemap.xml from the results. Make sure this is uploaded

Now you can log into Google Webmaster Tools and begin the Change of Address make sure that you have added your link to the sitemap.xml into the new profile. Google will now begin to make its necessary changes. This could take some time.

Is your analytics reporting?

If you have kept the same analytics account make sure you change the details so that it works with ok on the new domain.

If your site is an ecommerce one, do some test orders and see if the URL’s in your basket path have changed at all. Check your URL’s at every basket stage and make sure they match the basket stages in your Goal path funnels. Also check your final Goal URL’s. If they have changed then you’ll need to update all of your Goal points to reflect this change

Hello Webmasters

Now you need to pull out that list of backlinks you created and spend some time every day for the next few weeks contacting webmasters in that own the sites that you have heavyweight links on. Some sites are easy and have contact forms or even email addresses that you can use. Some sites can be harder. Firstly you need a name, if there is nothing on the site with a webmasters name you can check the code. Does he have a <meta name=”author” content=”Joe Bloggs” /> in there? If there is no name anywhere on the site pull up your favourite whois tool (I use Domain Tools) and tap the url in there. You should get sufficient information there to be able to jump on LinkedIn or something else that you use to find people.

I had quite a consistent message that I created. Part of it was that same for all sites and a section customised for that website. The intro went into the changes undergone and the custom part said something like “we have a link on your xyz page pointing to http://old-domain.com/dull-coins please can you update this to http://new-domain.com/new-coins”

Are your eyes peeled

You need to monitor everything daily now to make sure the transition goes smoothly. Keep a close eye on your webmaster tools accounts for 404′s, check your websites error logs. Look for anything odd in your analytics packages. Make sure your website is converting and sales are coming in.

PR / Newsletter

You need to tell the world of your new domain name change. If its just the domain change then it is a smaller Press Campaign and an update to your customers. If you did a site rebrand at the same time then you will obviously do a larger Press Campaign and a Newsletter. Most people won’t even spot the changes as you will have set up all your redirects correctly.

Finally keep monitoring EVERYTHING very closely to do with the site until you are comfortable that its all gone right. I was happy and comfortable after about a month and a half.  Some people say three months before it is all back to normal for you.

Some of my results

Unbelievably the process for me was so smooth I thought something was really wrong. My main ranking report has 16 big generic keywords in all of them are front page on Google. We did our switch on the 22nd December 2009. 2 days after the switch I started noticing some rankings appear for the new domain (shocked me). 5 days in and all of the rankings had moved over. Granted they were around 30 – 50 but they had moved over. 10 days in and the rankings had gone top 20 and by there they fluctuated for a while. Then one day on the 13th of January they got back to the positions they were before the switch +/-2 positions. A month later we seen a substantial increase in our rankings and they are now at better positions than before the domain switch. My personal thoughts are that the domain name we switched was twice the age of the old domain name, this must have had a bit of an impact with the rankings. Also my increased link building and quality directory submissions could have helped. As with everything in SEO there are a number of factors that have helped me get to a better position all of them together make for great success.

Things to remember

  • 301 301 301 Three Oh One did I mention 301? The most important thing here forgetting this or missing important pages will be very costly.
  • Remember to update your links on your website to point to your new url’s and page paths.
  • Try to contact as many webmasters as possible to amend those back links.
  • Test until you cannot test no more. If it all works brilliant and no broken links are on the test machine then it should be good to go.
  • NEVER forget your PPC campaigns. You will need to modify all of the display and target URL’s a big task if you manage a few hundred thousand keywords.
  • Tell all of your affiliates of the changes and where possible pass them and old to new URL list to make their life easier with the change. Also on any banners ensure you have the correct URL in them.
  • Never expire your old domain, it’s been yours for a few years. People like to type it in. It should be redirecting to the new domain homepage.

Because of the amount of research I did on this and the success of it all (so far so good) I will gladly answer any questions you may have on migration. The best way to grab my attention is through my Twitter account @shanejones {link to http://twitter.com/shanejones}

Shane Jones is an In House SEO working in for a leading Online Travel Agent in Manchester, England. You can contact him thought his site or follow him on twitter @shanejones{link to http://twitter.com/shanejones}

 Migrating to a New Domain and URL Structure   My Learnings
Shane Jones is an In House SEO working in for a leading Online Travel Agent in Manchester, England.
 Migrating to a New Domain and URL Structure   My Learnings

You Might Also Like

Comments are closed.

15 thoughts on “Migrating to a New Domain and URL Structure – My Learnings

  1. Solid piece, Shane! The only thing that I would mention is that after doing dozens of these massive migrations, I’ve come to realize that the simple act of creating server-side redirects (301s in most cases) will ensure that the value of all historical inbound links will pass through to the new corresponding domain/urls.

    That realization helps save a ton of time and effort, because it means that you don’t have to go through the trouble of reaching out to the webmasters that linked to you in the past.

    Think of 301 as soldering or welding…it literally welds the old, deprecated URL to the corresponding new URL.

    Now, there is still value in reaching out to webmasters that have provided inbound links in the past, because a major site migration is a good excuse to get the anchor text of existing inbound links updated to include relevant keywords and phrases, but that’s a different story all together…

    1. 301 redirects don’t pass the full value of your inbound links, you lose some…best way is to contact webmasters and change the links!

  2. Totally agree, without the 301 this process would probably never happen. But luckily we do have it. This is probably the biggest thing I have done in my In House SEO career. Was really refreshing to pull it off successfully too

  3. Great post Shane. I did a similar migration for a news site and wanted to contribute one other point: if the site has had a sitemove in the past chances are there are existing redirects in place. I’d update these redirects to point to the new URL locations so you’re not going through multiple 301′s (i.e. page A – B, not A – B – C…).

    Also use MajesticSEO to look at old backlinks from previous migrations (order by date discovered) and make sure these pass through a single 301 to new URLs.

  4. I’d also make sure your error pages have your Analytics tracking code attached, it can help you identify missed redirects (in Google Analytics you can do this by looking at your content by title report – but make sure the title makes it clear its an error page) as well as prevent self referrels.

  5. Solid tips here Shane, and you can tell they’re borne from experience. I’m planning a similar migration to a client site now and planning ahead is certainly the name of the game. It is a lot more tricky when you’re not an in-house as you only have a small amount of time to look over problems and issues. Here are a few things I’d add on to your list that I’ve found useful:

    - Testing 301s on a test server – use Xenu to crawl all existing URLs on old site, export to a page map and open up in Excel. Then do a find and replace to your test server domain and then reimport all the URLs back into Xenu – this should give you an idea if you’ve missed any 301s that should be in place – without waiting for the site to go live before you notice

    - Set up a 404 notification system like http://www.linkpatch.com (free for a single domain) – by adding a bit of code to your 404 page this will ping you an email every time an actual user hits a 404 page and tell you where they came from – bloody useful at quickly picking up cracks that you missed out

    - One of the issues that comes from a domain / URL migration is waiting for search engines to actually pick up the redirects – for lower priority pages this can take a while for the 301 to be picked up and updated in the SERPs. One technique I like the idea of is <a href="http://www.thegooglecache.com/white-hat-seo/xml-sitemap-assisted-redirects-advanced-white-hat-seo/"XML Sitemap Assisted Redirects but recently WMT has been rejecting XML sitemaps containing redirected URLs – but I think you could create a temporary HTML sitemap with historical URLs so that these are recrawled within the new site. Be warned that Yahoo is rubbish at picking up 301s – I’ve seen old domains showing up literally years after 301s have been put in place.

    - Have patience… I’ve seen dropouts occur as a result of broad sweep changes such as these but if you’re doing this right the results will come back, and often better than before. But the worst thing you can do is panic if this temporary sandbox effect happens.

    Hope those are helpful =)

  6. @jaamit some good extras there, am setting up link patch now. Really really could be useful as we still have a handful of 404′s coming through.

  7. hi shane that was really nice post. Mate i need little help
    I have dedicated server and my 50 sites are hosted on same. Now due to some reason my server is showing in blacklisted category due to email spamming which in house guys did last year. When we send mail it goes to spam
    Only issue is Will it create negative impact in google rankings

    I mean will google penalize our site rankings because of email spamming
    My ip is 88.208.216.153 and i check on many sites like http://www.mxtoolbox.com
    please help

  8. Excellent article, actually a wall hanger for sure. All sound statements, that sadly most people never even think of when making a change.

    Thanks again, your blog is now bookmarked :)

  9. Hi Shane,

    Great article that will hopefully help alleviate some of the fears people have with this. It will always be a daunting task but your experience is definitely one I’ve encountered, thankfully.

    I don’t think you mentioned 301′s enough though…

    Just one thought that might be relevant: You did this change slap bang in the middle of the Google caffeine roll-out. Aside from you being braver than me in that respect (needs must), it may have had a bearing on the improved rankings. Have you had any -ve results since?

    Again, thanks for sharing.

    James.

  10. Awesome post!

    These projects are *very* time-consuming, and getting all of the details right before pushing the ‘deploy’ button is absolutely crucial.

    One very important point about the 301 redirects, is that any request for any of the old URLs, both canonical and non-canonical, must pass through a single 301 redirect to arrive at the final, correct, canonical new URL for that content.

    That is, canonicalisation rules for old URLs, those that are correcting non-canonical requests within the old site, also need to be modified to instead redirect to the new canonical URL on the new site. Any and all ‘redirection chains’ must be avoided where at all possible.

    That is (for example) the non-www to www redirect on the old site has to be modified to redirect non-www requests on the old site directly to www on the new site. Additionally, you might need either new rules for redirecting www on the old site to www on the new site or you will modify the existing non-www to www rule on the old site to cope with both non-www and www requests on the old site. Finally, there are the essential new rules for redirecting non-www on the new site to www on the new site.

    That’s a simple example. The complexity can become almost mind-boggling, and making lists of URLs and looking for patterns that can be redirected in bulk using a low number of rules is crucial. Rule ordering is also important, both in preventing conflicts and redirection chains, as well as increasing the efficiency of operation.

  11. Well done Shane glad it all turned out due to your hard work and thorough approach thanks for sharing your insights will us here. Ive added this to my blogroll. Cheers MauriceCFlynn

  12. Great Article Shane. My question is something you didn’t touch on. We purchased a domain form another company because it aligns with our branding. We set it up to redirect to our current domain about six months ago. I heard this was a bad idea as we need to build relevant history before migrating to the new domain. what are some thoughts on this? Should we ignore that and make the switch when we are ready?