How To Run an SEO Split-Test (the Easy Way)

How to Run an SEO Split-Test (the Easy Way)

Google makes hundreds of changes to its algorithm every year.

As SEOs, you have two choices when it comes to dealing with them: make reactive changes to your website if it has been negatively impacted, or get proactive with split-testing to make sure it’s doing its job in the first place.

Statistical SEO split-testing means dividing pages of similar intent on your website into two groups, making a change to one of them, and monitoring the impact it has on a key SEO metric, such as organic traffic, click-through rate (CTR), or keyword ranking positions.

The result: small, but important tweaks to your website that improve it for users and help ensure that future algorithm updates are good news for your organic traffic, not bad. 

Let’s take a look at how you can run SEO split-tests without complex code, a data scientist team, or developer help. Here’s what you can expect from this guide:

1. Check Your Site Structure Includes Templatized Pages
2. Set a Hypothesis
3. Split Your Pages into Control and Variant Test Groups
4. Make the Change to the Variant Pages
5. Report on Your Test Results

1. Check Your Site Structure Includes Templatized Pages

The first step of conducting an SEO split-test is to check your site is large enough to ensure statistically significant results.

A test with too few pages or not enough monthly organic traffic will be unreliable. It will be nigh on impossible to tell whether the change you made actually had an impact. Other things, such as a slight increase in search demand to one category, could cause inflation. If the test wasn't successful, external factors could have made the impact.

So, what’s needed in order to run a statistically significant test? Сounts as a ‘large enough’ site? 

To run a reliable SEO split-test, your site should contain templatized pages that have the same repeatable format and similar intent across your site. Here are some examples:

• If you’re an e-commerce site, your category and product pages are templatized; 
• If you’re an affiliate site, you might have user-generated content pages that are templatized; 
• If you’re a directory site, you might have a large list of businesses or restaurants that are made up of templatized pages; 
• If you’re a stock photo site, all of your stock photo pages will have the same template; 
• If you’re a content or media site, your blog posts will all use the same template; and
• If you’re a marketplace site, you might have a list of vendor pages that all follow the same HTML/CSS template.

Your site should have a minimum of both of the following to make an SEO split-test worthwhile:

• 300 templatized pages (there are exceptions, read here)
• 30k monthly organic clicks per test group (100K last 100 days)

2. Set a Hypothesis

Next, decide what you want to test based on your SEO strategy and priorities.

There are a lot of SEO experiments you can use as a base for your split-test. Some examples include:

• Title tags and meta descriptions;
• Headings;
• Internal links;
• Content structure; and
• Alt text

20+ SEO Experiments to Get More Organic Traffic

Once you have chosen your focus area, use this framework to set a hypothesis: “We know [something], so we predict that [this will happen]. We’ll test this by [action] and measuring [metric].”

For example: “We know that customers use our free shipping options more than any other delivery method, so we predict that adding ‘free shipping’ to the meta tags will increase organic traffic. We’ll test this by adding ‘free shipping’ to our Variant group's meta description and measuring the difference in organic traffic.”

Before you start your test, it’s important to remember that statistical SEO testing is different from Conversion Rate Optimization (CRO), and user experience testing. 

With user experience testing, on the one hand, you make changes to on-page elements like calls-to-actions (CTAs), checkout buttons, and messaging to see how visitors’ behaviors change. With SEO split-testing, on the other hand, you optimize a listing to see how a search engine’s crawlers interpret a change to find better results for those users. 

The goal of SEO split-testing is to attract more organic traffic to a webpage, and not to increase your on-site conversion rate. It’s why most traditional A/B testing tools, such as Google Optimize, Optimizely, and A/B Tasty, can’t do SEO testing.

3. Split Your Pages into Control and Variant Test Groups

Once you have the hypothesis and list of pages to test, you need to divide them into two groups:

1. The Control group: The original pages, which will stay the same; and
2. The Experiment (Variant) group: The test pages on which the changes will be implemented

This saves you from creating different versions of the same page. You will make the changes to the pages in the Variant group, and compare them to those in the Control group at the end to see whether you had a positive, neutral, or negative impact. 

So, evenly divide pages with similar intent — like product/category or review pages. The user is visiting each page with a similar goal, be that buying an item or searching for businesses in their local area.

You can use filters to run tests on specific pages, too. For example: if you’re testing whether review Schema impacts CTR, you can exclude pages that are less than a month old. 

Manually splitting hundreds of pages can be an enormous task for a data science team; it involves exporting the data into a spreadsheet and weeding out irrelevant pages, such as noindex or non-canonical URLs, old content, and custom links that redirect to other pages. 

Even after you have your list, you will have to run smaller preliminary tests to check your Control and Variant groups are statistically similar.

The good news is that this tedious task can be made easy with SplitSignal. 

The tool does the hard work of segmenting pages by predicting the Variant group based on the Control pages using historical data from Google Search Console (GSC). All you need to do is connect Google Search Console to SplitSignal and, SplitSignal will automatically split them into Control and Variant groups for you.

4. Make the Change to the Variant Pages

Now’s the time to implement the change on your Variant test pages.

Remember that you will only be making changes to this group; the Control group of pages will stay unchanged so you can compare the two at the end.

URL-based experimentation is too difficult to do manually. You’ll need to create a ticket for your dev team before implementing any changes on the site. You’d also need to negotiate with them, and prove it’s worth prioritizing your SEO split-test above other tasks on their plate. That alone could take several months.

You could also build an in-house tool to do it for you (which will likely take up a sizable chunk of your annual SEO budget). 

The easiest way? SplitSignal. After the pages have been split, configure which change you want to make to the Variant group — such as adding “free shipping” to the title tag.

SplitSignal uses lightweight Javascript code (via Google Tag Manager) to make client-side changes to the pages in the Variant group. You won’t even need to bother with complex code to perform your URL-based experimentation.

A/B testing best practices apply here: you should only test one hypothesis so you can be confident that the positive or negative change was a direct result of the test and nothing else. 

Does This Count as Cloaking?

Cloaking happens when people and crawlers see different versions of one page.

This approach to SEO split-testing, however, doesn’t go against Google’s Webmaster Guidelines for cloaking. It applies changes to the page regardless of the viewer being a bot or a human, so both bots and users always see the same version of the Variant page.

5. Report on Your Test Results

It can take between 14 and 42 days to get reliable results from your statistical SEO split-test. 

The time it takes to get a result depends on how fast pages are indexed by Google, and whether the results are significant. Generally speaking, we suggest running tests in the 21 to 28 day range.

You will see the real vs. predicted traffic of pages in your Variant group in your SplitSignal dashboard, so there’s no need for running complex calculations. The prediction for Variant traffic is made using the CausalImpact model and based on the Control traffic. It is then compared to the real traffic to the unchanged pages. 

If the difference is big enough, then we consider the test statistically significant.

Take a look at the three examples below. The orange line is the Variant group; the blue line is the Control group.

Variant pages in Test A performed significantly better than those in the Control group. 

The reverse was true in Test B, however: the Variant group actually performed worse than the unchanged Control group.

Finally, Test C showed no significant difference between the groups.

6. Use the Test Data to Fine-Tune Your SEO Strategy

Once you get your SEO test results, you’ll have a solid understanding of which tweaks you should continue sitewide. 

For example: if your test concluded that adding "free shipping" to a meta title lifted organic traffic by 10%, it makes sense to roll out the change on more templatized pages. 

The key to SEO split-testing is to work on previous results. So, if the meta title test was positive, make your sample sizes larger. 

The risk of rolling out the change to higher value pages is low since your confidence from the initial test is high. You already proved the hypothesis on a small test group. Rolling out similar changes on a wider group of similar intent pages is likely to yield similar positive (or negative) results.

Ready to Start SEO Split-Testing?

There’s no doubt that SEO split-testing is crucial for understanding how Google interprets (and ranks) your website. By testing small tweaks to page groups, you’ll begin to uncover which subtle tweaks move the needle for your SEO strategy.

The good news? You can easily run statistical SEO split-tests at scale with SplitSignal, and spot whether your tests are successful. You don’t have to ask your data science team to split pages, your development team to implement them, nor do complex equations to decide statistical significance. 

The data will be interpreted; you’ll be able to see the end result (and how we arrived there).

The best part? All changes are made through lightweight code that has no significant effect on Core Web Vitals. 

Granted, the tweaks you’re testing might seem small. But a small change from 10K to 10M product pages could lead to a 10% to 20% lift in organic traffic to those pages, which would be worth millions of dollars. 

( Exclusive Concepts, for example, saw a 21% increase in organic clicks on variant pages. That change will increase sitewide revenue by 5% over the next 12 months.) Learn how to prove the ROI of SEO split-testing.

Start running your statistical SEO split-test today with SplitSignal.