amazon ppc and split testing

How to Use Amazon PPC to Power Your Split Tests

andrew browne Split Testing Best Practices 2 Comments

Sharing is caring!

In order to run an effective split test, you need to drive traffic to your listings. If you are just starting out or you are having difficulty ranking organically, this may be hard to do.

Amazon PPC can help you drive traffic to your listing, which in turn can help Splitly reach statistical significance when you are running a split test.


Amazon Pay Per Click Can Help Drive Traffic and Conversions to Your Listings

In a previous article, I have said that I don’t like to mix Amazon PPC while doing a split test. Sorry to be a walking contradiction 😉

However, there are certain situations where it may be helpful to use Amazon PPC along with Splitly. Mainly, if you don’t have a lot of traffic going to your listing, you might not be able to run an effective split test. This is because we need to reach statistical significance before Splitly can accurately tell what caused a change in your sales metrics.

So this is where your paid ads can help boost both your sales and your split test significance.


Why Do We Need to Drive Traffic to Our Listings in Order to Split Test?

In order to determine what is causing your increase or decrease in Amazon sales metrics, either the variant you are testing or real world factors, we need to use statistical significance.

Statistical Significance

Statistical significance is how confident we are that the thing you are testing is the cause of any increase or decrease in your Amazon sales metrics, and not other real world factors.


For Splitly, we show statistical significance as a percent. The higher the percent, the more we are sure that any changes in your Amazon sales metrics are from the variant you are testing.

For example, lets say you run a test on one of your product images. From this, Splitly finds that there is a 98% statistical significance that this image change caused an increase in your sales. This means that we are 98% sure this change caused the increase and not outside factors.

In order to get statistical significance, we look at your past sales data. This is to see what your normal sales data has been and then compare it to your sales data while you split test.

If we see an increase or decrease in your sales metrics during your variant test, we can compare it to your past data to see if this increase or decrease was due to your variant or just a normal occurrence.

Volatility & Sales Volume

We also factor in past volatility. Volatility is an extreme increase or decrease in sales, usually seen as spikes and dips in your sales data. If you do not have a large amount of sales history, we consider your sales history very volatile, because we don’t have a lot of data to compare.

Therefore, it is harder to calculate statistical significance when you don’t have a large amount of sales history, and you should consider driving more traffic to your listing in order to get a more robust sales history.

In addition, increasing sales volume will help us determine statistical significance by allowing us to test the variant and non-variant more times and to factor out volatility. The more times you are able to test a variant, the more likely the outcome is correct because it reduces the chance that the outcome was due to randomness.

For example, if you flip a coin 5 times, it is not hard to imagine getting tails 4 times. However, if you flip a coin 1,000 times, you are more likely to see a 50% split between heads and tails. This is because the more times you do something, the less randomness will affect the results.

Therefore, the more traffic we have to your listings, the more we can test the variants, and the less randomness will affect the results. Amazon pay-per-click may be a good way to drive more traffic to your listing.


When Can You Use Splitly and Amazon PPC Together?

If the current traffic to your listings are very low or you are just starting out, you may need to drive paid traffic to your listing in order to get enough eyes on your listing to run a split test. One of the fastest and easiest ways to do this is with Amazon PPC.

As the ads start to drive traffic to your website, you can start to use Splitly to help split test your listings in order to make them fully optimized.

Not so fast though… if you are going to run a split test in conjunction with Amazon pay per click, there are a few rules you must follow. Here’s a top level overview of them:

golden rules of running Amazon ppc and split tests simultaneously


Let’s dig a little deeper into the best practices.

You can use Splitly in conjunction with Amazon PPC in two ways:

  1. Use PPC to boost more sales, which in turn can help you get more reviews and hopefully boost your rank. This increase in rank should help increase organic traffic. Once organic traffic increases, turn on Splitly and optimize your listing to either try to increase more sessions or increase conversions. OR
  2. Use Splitly and Amazon PPC at the same time(!)

You will need to determine how you approach this dependent on your own situation.

How To Run Amazon PPC ads and Run Split Tests Simultaneously

As we have said, volatility has an impact on statistical significance, which makes split test results harder to determine. So how can we run PPC ads and reduce unnecessary volatility? Let’s take a look at the rules I mentioned above in more detail.

Rule #1

let ads run for a few weeks clock icon

Let the ads run a few weeks before you start split testing so your sales metrics normalize!

When you first turn on Amazon PPC you are likely to see some changes in your sales metrics including a spike in sessions, impressions and conversions. If you turn on PPC right before you split test, Splitly might not be able to distinguish which caused the spike in these metrics. Was it from PPC or the variant you are testing? If you have been running Amazon pay per click for at least a few weeks, your sales metrics will start to normalize, and Splitly will be more likely to tell which caused the change in sales metrics.

Rule #2

don't turn on in middle of a test calendar icon

Don’t turn on Amazon PPC in the middle of a split test

Similarly, if you turn on Amazon PPC while split testing, you are likely to see a spike in your sales metrics. This spike in sales metrics might be caused by the variant you are testing. However, it might be caused by turning on Amazon PPC. Doing this in the middle of a split test will make it extremely hard for Splitly to determine the cause.

Rule #3

don't change your keywords during a split test search icon

Don’t change the keywords you are advertising while split testing

Most Amazon PL sellers I know are extremely data driven. They often like to take the data they get from Amazon PPC ads and try to optimize their ad campaigns. This is a great strategy. However, changing your keywords to try to optimize your ad campaign may cause an increase or decrease in your sales metrics. Again, this increase / decreases is seen as volatility, making it hard for Splitly to determine the cause of of the increase / decrease. If you plan on making a change to the keywords you are targeting, try doing it well before you split test, so your sales volatility can normalize. Either that, or wait until after you split test.

Rule #4

set your budget and bid price early icon

Set your budget and bid price well before testing and stick with it.

Another thing which might cause a spike in your sales metrics is when you increase your bid price and / or budget. This spike might cause some confusion, as it will make it harder for Splitly to determine whether this spike was due to the change in your variant or the change in your PPC pricing strategy. Therefore, if you are going to make a change in your PPC pricing strategy, do so a few weeks before or after you conduct your split test. Make sure to give enough time to let your sales metrics settle.



So there we have it. Amazon PPC can help you drive traffic to your listings, which in turn can help you increase your chances of reaching statistical significance while conducting a split test. Not only that, running Amazon PPC ads is a highly recommended tactic for many reasons, which our friends at Jungle Scout have written several insightful articles about.

If your listing lacks traffic you may want to consider this. If you do, you need to decide whether to run your ads in conjunction with PPC tests or not, and follow the best practice guidelines in this article.


Have you had any successes running Amazon PPC and Splitly A/B experiments? I’d love to hear about it in the comments. Or if you have any questions, feel free to say hello!

Sharing is caring!

Comments 2

  1. Hello there Andrew Browne !!

    Extraordinary bits of knowledge Andy. This is truly useful to know before beginning with split testing. I might want to share this You have connected here some more educational and expected angle for me

    A debt of gratitude is in order for sharing such perspective,

Leave a Reply

Your email address will not be published. Required fields are marked *