Using Artificial Intelligence to Solve #SEO

SMS Text
Using Artificial Intelligence to Solve #SEO | SEJ

I recently wrote about how to statistically model any given set of search results, which I hope gives marketing professionals a glimpse into how rapidly the SEO industry is currently changing in 2015. In that article, I had mentioned that the search engine model should be able to “self-calibrate”, or take its algorithms and weightings of those algorithms, and correlate the modeled data against real-world data from public search engines, to find a precise search engine modeling of any environment.

But taking thousands of parameters and trying to find the best combination of those that can curve fit search engine results is what we in computer science call a NP-Hard problem. It’s astronomically expensive in terms of computational processing. It’s really hard.

So how can we accomplish this task of self-calibrating a search engine model? Well, it turns out that we will turn to the birds — yes, birds — to solve this incredibly hard problem.

Full Disclosure: I am the CTO of MarketBrew, a company that uses artificial intelligence to develop and host a SaaS-based commercial search engine model.

Particle Swarm Optimization

I have always been a fan of huge problems. This one is no different, and it just so happens that huge problems comes with awesome solutions. I turn your attention to one such solution: Particle swarm optimization (PSO), which is an artificial intelligence (AI) technique that was first published in 1995 as a model of social behavior. The technique is actually modeled on the concept of bird flocking.

Using Artificial Intelligence to Solve #SEO | SEJ

An Example Performance Landscape of Particle Swarm Optimization in Action

The optimization is really quite remarkable. In fact, all of our rules-based algorithms that we have invented to-date still cannot be used to find approximate solutions to extremely difficult or impossible numeric maximization and minimization problems. Yet, using a simple model of how birds flock can get you an answer within a fraction of time. We have heard the gloom and doom news about how AI might take over the world some day, but in this case, AI helps us solve a most amazing problem.

I actually have been involved with a number of Swarm Intelligence projects throughout my career. In February 1998, I worked as a communications engineer on the Millibot Project, formerly known as the Cyberscout Project, a project utilized by the United States Marines. The Cyperscout was basically a legion of tiny little robots that could be dispersed into a building and provide instant coverage throughout that building. The ability of the robots to communicate and relay information between one another, allowed the “swarm” of robots to act as one, effectively turning a very tedious task of searching an entire building into a leisurely stroll down one hallway (most of these tiny robots each had to travel a only few yards total).

Why Does It Work?

The really cool thing about PSO is that it doesn’t make any assumption about the problem you are solving. It is a cross between a rules-based algorithm that attempts to converge on a solution, and an AI-like neural network that attempts to explore the problem space. So, the algorithm is a tradeoff of exploratory behavior vs. exploitative behavior.

Without the exploratory nature of this optimization approach, the algorithm would certainly converge on what statisticians like to call a “local maxima” (a solution that appears to be optimal, but is not optimal).

First, you start with a number of “flocks” or guesses. In a search engine model, this may be the different weightings of scoring algorithms. For instance, with 7 different inputs, you would start with at least 7 guesses at these weightings. A specific guess would simply be an array of 7 different weightings.

Market Brew Self Calibration

An Example Set of Search Engine Model Input Factors

The idea behind PSO is that each of these guesses should be spaced apart so that none of them are close to each other. Without getting into 7-dimensional calculations, you can use a number of techniques to ensure your starting points are optimal.

Then, you begin to “evolve” the guesses. The evolutionary process tries to mimic what a flock of birds would do when presented with a cache of food nearby. One of the random guesses (flocks) will be closer than the others, and other guesses will adjust their next guess based on this global information.

I’ve included a great visualization on what this process looks like (below).

A Visualization of How Particle Swarm Optimization Works


Fortunately, there are a number of implementations out there in various coding languages. The great thing about PSO is that it is very easy to implement! There are very few tuning parameters (a good sign of a strong algorithm), and very few drawbacks.

Depending on your problem, your implementation may get trapped into a local minimum (non-optimal solution). You can easily fix this by introducing a neighborhood topology, which effectively limits the feedback loop to the best of the nearby guesses instead of the best global guess so far.

The bulk of your work will be designing the “fitness function”, or the scoring algorithm that you will use to determine how close you are to your target correlation. In our case, with SEO, we want to correlate against a known entity, like Google’s US results. You could just as easily swap this out with any version of any search engine.

Using Particle Swarm Optimization to Correlate a Search Engine Model

Using Particle Swarm Optimization to Correlate a Search Engine Model

Once you have your scoring system in place, your PSO algorithm will attempt to maximize that score across potentially trillions of combinations. The scoring system can be as simple as doing a pearson correlation between your search engine model and the public search results, or it can get as complex as taking those correlations and additionally assigning points to specific scenarios (e.g. 10 points for matching #1, 5 points for matching #2, etc…).

The Problem With Correlating Against A Black Box

Recently, there have been many SEOs out there trying to correlate against Google’s “black box”. I applaud this effort, but it is severely misguided. Allow me to explain.

First, correlation does not always mean causation. Especially if the inputs to your black box are not close to the outputs. Let’s show some examples. First, an example where inputs are very close to their respective outputs: an ice cream truck business. When there are warmer temperatures, people buy more ice cream. It is easy to see that one input (the temperature) is closely tied to the output (ice cream).

Unfortunately, most SEOs do not enjoy any statistical closeness between their optimzations (inputs) and their respective search engine results (outputs).

The Anatomy af a Search Engine

Their inputs, or optimizations, are before even the crawling components of the search engine. In fact, typical optimizations have to go through four layers: crawling, indexing, scoring, and finally the real-time query layer. Trying to correlate this way is fools gold.

In fact, Google actually introduces a significant noise factor, similar to how the U.S. government introduced noise to its GPS constellation, so civilians would not be able to get military-grade accuracy. It’s called the real-time query layer. The query layer is currently acting as a major deterrent for SEO correlation tactics.

Using Artificial Intelligence to Solve #SEO | SEJ

The Noisy Query Layer Of A Search Engine

An analogy I always use is that of a garden hose. At the scoring layer of a search engine, you have the brand’s view of what is going on. The water coming out of the hose is organized and predictable — that is, you can change things and accurately predict the resulting change in water flow (search results).

In our analogy, the query layer spreads this water (search results) into millions of droplets (variations of search results), depending on the user. Most of the algorithm changes today are occurring on the query layer, in order to produce more variations of search results for the same number of users. Google’s Hummingbird algorithm is an example of one of these. Shifts in the query layer allow search engines to generate more marketplaces for their pay-per-click ads.

Using Artificial Intelligence to Solve #SEO | SEJ

Finding Causation Through Correlation Can be Fools Gold

The query layer is the user’s view of what is going on, not the brand’s. Therefore, correlations found this way will very rarely mean causation. And this is assuming that you are using one tool to source and model your data. Typically, SEOs will use a number of data inputs for their modeling, which only increases this noise and decreases the chances of finding causation.

How To Find Causation In SEO

The trick to getting correlation to work for a search engine model is to tighten the inputs and outputs significantly. The inputs, or the variables, must be located at or above the scoring layer in the search engine model. How do we do this? We have to break down the search engine black box into its primary components. We have to build the search engine model from the ground up.

The outputs are even harder to optimize: we already discussed why the public search engines have a tremendous amount of noise due to the real-time query layer creating millions of variations due to the user. At a minimum, we will need to make our search engine model output results that are BEFORE the typical query layer variations. This will ensure that at least one side of the comparison is stable.

If we build the search engine model from the ground up, we would then be able to display the search results coming directly out of the scoring layer instead of the query layer, which gives us a more stable and precise relationship between these inputs and outputs that we are trying to correlate. Now, with these tight and significant relationships between inputs and outputs, correlation means causation. When we boost the importance of one input, it gives us a direct connection to the results we see. We can then do the typical SEO analysis to determine what optimizations would be beneficial in the depicted search engine model.


As a fan of artificial intelligence, I am always very excited when something so simple in nature can lead to scientific discoveries or technological breakthroughs. Having a search engine model that allows us to transparently connect scoring inputs with non-personalized search results gives us the ability to connect correlation with causation.

Add in the Particle Swarm Optimization and you have a technical breakthrough: the self-calibrating search engine model.


Image Credits

Featured Image: Willyam Bradberry via Shutterstock
Image #1: Particle Swarm Optimization Chart from Wikipedia Commons.
All other screenshots and diagrams taken from MarketBrew’s Predictive SEO system in 2015.

Scott Stouffer

Scott Stouffer

Co-Founder + CTO at MarketBrew
Scott Stouffer is a Co-Founder and CTO at MarketBrew, a predictive platform for digital marketers. MarketBrew's predictive engine simulates any brand's competitive landscape online, prescribes top website optimizations and predicts how these changes will impact search engine ranking, 60 days before they happen. Scott has a masters degree in both Electrical and Computer Engineering from Carnegie Mellon University and is also an instrument-rated pilot.
Scott Stouffer
Scott Stouffer
Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • Sid Silhouette

    Excellent article. I have found that an optimizer will need several tools to analyze data. You can’t just trust one.

    • Scott Stouffer

      Sid, thanks! Yes, in a search engine model, you will want multiple sources of data to confirm your findings. That being said, in the world of signal/noise, adding two noisy signals does not cancel out. In fact, this adds extra noise. And the more disparate tools that you use (disparate = drawing from different original data sources), the more noise you will induce.

      So you need to choose the confirming model data wisely ūüôā

  • Jonathan Wilhelm

    Particle Swarm Optimization sounds like something I would like to learn more about. Anyone know any good books on the subject? Are there any groups in the Silicon Valley that are worth checking out?

  • Manish thakur

    Great article but more technical

  • Matt LaClear

    There may be thousands of SEO professionals, at this moment, with their hand on the light switch and wondering how to turn it on. It would be great if you could follow up with a simple application guide. Thanks for a major share Scott..

    • Scott Stouffer

      Thanks Matt. Will have lots more to share in upcoming articles! I’m trying to make things as technology/brand agnostic as possible so all of you understand the concepts first, which is important.

      If you are in a highly competitive environment and are interested immediately in using this technology, MarketBrew sells a commercial search engine model that can be targeted against 20+ different search engine environments.

  • Umer

    Great Article .. Found really useful info Thanks scott

  • David Carmichael

    Thanks for a fascinating article.

    I have a book on my shelf titled “Artificial Intelligence – A Modern Approach”, published in 1995. Needless to say, it’s no longer modern, although it provides a first rate introduction to the subject.

    Only recently did I bother to look up what the author, one Peter Norvig, is doing twenty years on . . . Seems he’s Director of Research at Google!

    • Scott Stouffer


      I’m glad to connect. I’ve studied Peter for quite some time, did you know he was NASA’s senior computer scientist?! Look his site up, it’s like going back to 2005, when it was all about content and links!

      I actually started researching how to build a search engine model back in 2007, so I was actually quite inspired by his team’s work.

  • Nate Dame

    Interesting article – very technical and difficult to take on. The proof is in the pudding – case studies of your process applied might go much further in making me/others a believer.

    But how do you possibly cope with user action inputs, such as click thru rates and bounce rates, and their impact on rankings? There have to be dozens of ranking factors that we could never access, making any analysis we might hope to make (even when AI-based) simply correlation… no?

    • Scott Stouffer

      Hi Nate! The company that I founded, MarketBrew, has indeed been able to test and verify that this works (over the last 8 years, with the last 2 years getting extremely accurate because of Particle Swarm Optimization!).

      My current plan is to share a number of case studies over 2015 and 2016, not just to show that it works (it works extremely well) but also to explain things in practical, pragmatic actions, which are sometimes easier to digest than bunches of theoretical data.

      Now to answer your very insightful questions:

      1) How do you possibly cope with user action inputs, such as click thru rates and bounce rates, and their impact on rankings?

      You can safely separate the Query Layer (which these types of things: click-through rate, etc… are a part of) from the Scoring Layer in a Search Engine Model. I wrote a very specific white paper on this topic ( which explains this better.

      2) There have to be dozens of ranking factors that we could never access, making any analysis we might hope to make (even when AI-based) simply correlation… no?

      This approach ignores trying to perfectly match each algorithm (which is a pretty insane approach, even if you live at 1600 Amphitheatre Parkway). Instead, it takes a generic model (with sets of generic algorithms the represent core foundational techniques that all modern search engines use), and uses its ability to “machine learn” the AI-based search engine environment that it is targeting. In this way, the model effectively skips over the requirement of needing every single algorithm/tweak to be represented.

      Here is an example: let’s say you wanted to target “Austin Car Dealers” on Google U.S. You could self-calibrate the model to this local environment and have a good idea of how each ranking factor drives that local search engine market (results). But let’s also say that Google engineers decide one day (or the AI-based algorithms decide) that META Title should now be more important, because it thinks that will be less spammy for that local market. Your model should then attempt to detect that it is out of calibration, re-calibrate, and document what changed in the model.

      With this approach, you didn’t know which exact algorithm changed, but you were able to obtain info that it was a META Title algorithm that caused that ranking factor to change. All of your new decisions would now be made off of this new information in the model. You effectively “machine learned” the machine learner, without prior knowledge of what changed.

      I hope this helps!