Wish your product listings could rank better on search engines?
Feel like your success is as limited as your crawl budget?
Effectively ranking thousands of products, hundreds of categories, and millions of links requires a level of organization that can sometimes feel out of reach.
This is particularly true when the performance of your ecommerce site depends on a limited crawl budget or disconnected teams.
For large ecommerce sites, it’s an immense challenge.
In this post, you’ll find proven solutions for some of the most persistent technical SEO issues plaguing sites like yours. You’ll learn how to tackle issues around crawl budget, site architecture, internal linking, and more that are holding your site’s performance back.
Let’s get to it.
1. Crawl Budget Is Often Too Limited To Provide Actionable Insights
Growing your ecommerce business is great, but it can lead to a massive volume of pages and a disorganized, outdated website structure.
Your company’s incredible growth has likely led to:
- Extensive SEO crawl budget needs.
- Lengthy crawling processes.
- High crawl budget waste from easily-missed, outdated content, such as orphan and zombie pages, that no longer need to be crawled or indexed.
- Difficult-to-follow reports filled with repetitive fundamental technical errors on millions of pages.
- Incomplete and segmented crawl data, or partial crawls.
Trying to solve SEO problems using partial crawls isn’t a great idea; you won’t be able to locate all the errors, causing you to make SEO decisions that may do more harm than good.
Whether your crawl budget limitations are from website size or desktop-based crawling tools, you need a solution that allows you to review and understand your full website, as a whole — with no limits.
The Solution: Use Raw Logs Instead Of Crawl Reports
To overcome the issue of slow, limited crawl budgets, we recommend using raw logs instead of crawl reports.
Raw logs give you the power to:
- Monitor crawling, indexation, and detailed content analysis at a more reasonable price.
- Understand which pages are impacting your crawl budget and optimize accordingly.
- Eliminate critical errors right after a product update.
- Allow you to fix issues before Google bots discover them.
- Quickly identify pages with 2XX, 3XX, 4XX, and 5XX status codes.
Using a raw log tool also gives you the exact picture of a site’s SEO efficiency.
You’ll be able to pull reports that show the number of pages in the site structure, the pages getting search bot visits, and the pages getting impressions in SERPs.
This gives you a clearer picture of where structure and crawling issues occur, at any depth.
For example, we can see there are more than 4 million pages in the site structure above.
Only 725,161 are visited by search bots.
And only 29,277 of these pages are ranked and getting impressions.
24,189,025 pages visited by search bots that aren’t even part of the site structure.
What a missed opportunity!
How To Discover & Solve SEO Crawl Issues Faster With Raw Logs
Implement a no-limit SEO analysis tool that can crawl full websites of any size and structure.
Blazing fast, JetOctopus can crawl up to 250 pages per second or 50,000 pages in 5 minutes, in order to help you understand how your crawl budget is affected.
- Create an account at JetOctopus.
- Access the Impact section.
- Evaluate your Crawl Ratio and missed pages.
In seconds, you can measure the percentage of SEO-effective pages and know how to improve the situation.
Our Log Analyzer tracks crawl budget, zombie and orphan pages, accessibility errors, areas of crawl deficiency, bot behavior by distance from the index by content size, inbound links, most active pages, and more.
With its effective visual representation, you can boost indexability while optimizing the crawl budget.
Crawl budget optimization is central to any SEO effort and even more so for large websites. Here are a few points to help you get started.
Identify whether your crawl budget is being wasted.
Log file analysis can help you identify the reasons for crawl budget waste.
Visit the ‘Log File Analysis’ section to determine this.
Get rid of error pages.
Review the site’s crawl through log file analysis to find pages that may have 301, 400, or 500 errors.
Improve crawl efficiency.
Use SEO crawl and log file data to determine the disparities between the crawled and indexed pages. Consider the following to improve crawl efficiency.
- Make sure the GSC parameter setting is up to date.
- Check if any important pages are included as non-indexable pages. The data in log files will help you locate them.
- Add disallow paths in the robots.txt file to save your crawl budget for priority pages.
- Add relevant noindex and canonical tags to indicate their level of importance to the search bots. However, noindex tags do not work well in the case of multimedia resources, namely videos and PDF files. In such cases, use robots.txt.
- Look for disallowed pages being crawled by search bots.
2. Managing A Massive Internal Linking Structure Can Be Complicated
Internal linking is one of the best ways you can inform Google of what exists on your website.
When you create links to your products from pages on your site, you give Google a clear path to crawl in order to rank your pages.
Google’s crawlers use a website’s internal linking structure and the anchor text to derive contextual meaning and discover other pages on the site.
However, creating a crawl-friendly internal linking structure is tough for large-scale websites.
Keeping up with internally linked products that constantly go in and out of stock isn’t always sustainable on a large ecommerce site.
You need a way to see where deadends happen during a Google crawl.
Why Internal Linking Structure Matters
Google relies on internal linking to help it understand how visitors can quickly and easily navigate through the website.
If your homepage ranks well for a specific keyword, internal links help in distributing PageRank to other, more focused pages throughout the site. This helps those linked pages rank higher.
The Solution: Find Crawl Dead-Ends With Interlinking Structure Efficiency Tools
Our Interlinking Structure Efficiency solves this issue by giving you a clear view of your site’s internal linking health.
- On the dashboard, go to Ideas -> Structure Efficiency.
- This screenshot shows the list of directories that are present on the website, the pages in this directory, the percentage of indexable pages, the average number of internal links to a page within this directory, the bot’s behavior here, SERP impressions, and clicks. It clearly reflects SEO efficiency by directories to analyze and multiply the positive experiments.
Check out how our client DOM.RIA Doubled Their Googlebot Visits by experimenting with it.
- Crawlability: JS content limits a crawler’s ability to navigate the website page by page, impacting its indexability.
- Obtainability: Though a crawler will read a JS page, it cannot figure out what the content pertains to. Thus, they will not be able to rank the pages for the relevant keywords.
As a result, ecommerce webmasters cannot determine which pages are rendered and which aren’t.
It also shows the JS errors.
Here’s how to put this feature to work for you:
- Go to JS Performance in the Crawler tab.
4. Few Tools Offer In-Depth Insights For Large Websites
Core Web Vitals and Page Speed are significant technical SEO metrics to be monitored. However, few tools track these page by page.
The Solution: Use One Tool That Gives You True Priority Tasks
Unlike other crawlers, JetOctopus’ functionality is not just limited to being a SaaS Crawler or Log Analyzer. It’s a comprehensive content and SEO audit toolkit that ensures website metrics are in line with your goals.
Plus, it allows you to stay on top of the common technical SEO issues faced by large websites.
Prioritize Global Issues That Need Attention
With this feature, you can focus on what’s most pressing.
Use Data Visualization To Your Advantage
Use preset reports, automatic data overlapping, and vivid charts with valuable SEO insights to reap the most valuable insights in seconds.
You can see visualized data from Google Search Console (GSC), giving you a better understanding of what GSC is telling you so you can draw the right conclusions.
See how the Impact section offers clear insights into bot behavior within seconds:
Analyze More Valuable Data in Dynamics
If you’re only watching general trends in GSC, you may be leaving money on the table.
Dynamics enables you to:
- Watch the percentage change in impressions, clicks, CTR, active pages, and ranking in dynamics.
- Find out which URLs have grown or dropped over time. Find the reason and either fix the situation or multiply the positive experiments to the rest of the commercial pages.
- You can do all this without touching Excel. All charts shared are preset and clickable. Plus, you can add Google Analytics to gain deeper insights.
- You can also do a detailed keyword analysis with GSC to identify new keywords that are just starting to get impressions, lost keywords that have stopped getting impressions, and zero-click keywords with impressions but no clicks. You’ll see winners, and losers by impressions, clicks, CTR, and positions, as well.
5. Keeping Multiple Development & Product Teams Aligned With SEO
You know the drill: Three teams decided to update the UI of the full site without consulting you or your SEO team.
Two marketing teams decided to migrate a full cluster of pages without informing the rest of the company.
Your on-page SEO team decided to update its landing page templates without updating the schema markup.
By the time your technical SEO team detects it, it’s too late. Their UI updates end up harming the SEO of several pages or even the whole cluster of pages.
Resources are wasted, and getting back the lost traffic won’t be easy.
How can you find out about website changes, before Google does, and resolve them?
What’s the best way to keep teams running towards the same goal?
The Solution: Align Your Teams’ Goals & Get SEO Alerts That Tell You Before Google Finds Out
- Set up real-time alerts that let you know exactly when an SEO issue has occurred.
- Compile your business’s frequently occurring issues.
- Create and implement an inter-departmental strategy to keep all players involved.
- Leverage a project management tool to track checkpoints, ensuring that everyone is informed of important dates.
Sound like a lot of work? These issues can be easily solved with alerts, as shown below.
- Go to the Alerts tab and click on Overview.
- Now, you have a complete overview of the errors that may affect your site’s online performance.
Here’s how this feature can help.
The Easy Way To Configure SEO Alerts
Alerts give you control, SEO safety, and the opportunity to eliminate errors and fix them before the bots come to your website.
You will receive timely updates sent via email, SMS, messenger, and more.
- Sign up and configure your account and automatically receive SEO alerts.
That’s it. Once you’re setup, you’ll receive notifications on:
- Crawl budget changes.
- Index page down.
- Txt non-200 status.
- Increasing of pages with non-200 status code.
- An increasing amount of slow pages, >2 sec.
- Not working shopping card.
- And a lot of others.
Monitor Multiple Teams’ SEO Experiments
For large ecommerce sites, on-page SEO should be tracked regularly as small issues may snowball within no time.
Watch the dynamics of small SEO experiments, and take advantage of Log File Analyzer’s ability to provide a bulletproof argument to support SEO strategies and experiments.
This acts as a foolproof check for SEO efficiency.
If the outcomes of the experiments are satisfactory, you can easily apply them to the whole website.
The Easy Way To Create An Inter-Departmental Strategy
Use the segmentation tool to divide large amounts of website content into smaller and logical segments to be analyzed effectively.
Within moments, you’ll be able to:
- Do a quick analysis of SEO issues and identify the next action steps.
- Experiment with different complex segments (get data from different data sets in one selection) to find new ways to improve organic performance and conversions.
- Reduce manual work and errors, common with large website SEO.
Our segmentation tool crawls and segments ecommerce sites using different datasets and easy-to-use filters. You can join several datasets like GSC, BotLogs, Ahrefs, and crawl data to get a more detailed view of the site performance.
In this All Pages Crawl report, for example, you can see 1,662,633 crawled pages with 1,647,013 having critical issues and 1,633,039 having warning issues.
Notice what happens when you segment them with pages with unoptimized meta descriptions.
You see 184,910 pages, of which 171,173 pages are having critical issues and 166,348 pages with warning issues.
Why Is JetOctopus The Best Choice For Large Ecommerce Sites?
If you want to make your technical SEO program simpler, better visualized, and just more doable, we’ve got you.
Don’t bloat your workstream with dozens of tools when you can use one lightning-fast, comprehensive, SEO tool do it all.
JetOctopus’ intuitive UI makes it easy to navigate different sections of the report. The easy-to-use tool segments each part of the website like landing pages, blog sections, product pages, and more on its dashboard.
Affordable and Cost-Effective
JetOctopus is a high-tech SEO platform that’s affordable to all (not just enterprises).
Most crawlers do not allow users to completely crawl their client sites because of project limits.
JetOctopus understands that ecommerce websites are large; hence, we offer the power of ‘crawling without project limits’ at a reasonable price.
That means half-yearly or annual payments, as well as additional discounts(10-20%).
Choose your package and let’s get started!
The opinions expressed in this article are the sponsor's own.