Split Test Analysis & Amazon Sales: Know What Drives Your Private Label Sales

andrew browne Uncategorized 2 Comments

Sharing is caring!


Our sales may increase or decrease due to simple daily randomness.


In order to provide you with accurate information on why your sales/sessions are increasing or decreasing, we at Splitly use statistical significance.


Statistical significance is our way of telling you how confident we are that the variables you are testing caused the changes in your sales/sessions.


The higher your statistical significance, the more positive we are that the variables you tested caused the changes in your sales statistics.


There are ways to increase your chances of getting a high statistical significance, which include

  • testing big changes to your listings
  • avoiding small changes
  • continuing with your normal ad and promotion strategy
  • and testing one variable at a time


If you are getting a low statistical significance, try to run your tests for a longer period of time. Also try to make bigger changes to your listings.


The World is Full of Randomness


In one of my favorite movie scenes of all time, Dr. Ian Malcolm explains to Ellie Sattler what chaos theory is in Jurassic Park, before the dinosaurs began to run amok. The conversation went like this:

MALCOLM: [Chaos Theory] deals with unpredictability in complex systems. The shorthand is the Butterfly Effect. A butterfly can flap its wings in Peking and in Central Park you get rain instead of sunshine. . . .

MALCOLM: Give me that glass of water. We are going to conduct an experiment. It should be still, the car is bouncing up and down, but that’s ok, it’s just an example.

MALCOLM : Now, put your hand flat like a hieroglyphic. Now, let’s say a drop of water falls on your hand. Which way is the drop going to roll off.

MALCOLM: Off which finger or the thumb, what would you say?

ELLIE: Thumb, I’d say.

MALCOLM: Aha, ok. Now freeze your hand, freeze you hand, don’t move. I’m going to do the same thing, start with the same place again. Which way is it going to roll off?

ELLIE: Let’s say, back the same way.

MALCOLM: It changed. Why? Because tiny variations, the orientation of hairs on your hand- –

MALCOLM: – – the amount of blood distending your vessels, imperfections in the skin – –

ELLIE: Imperfections in the skin?

MALCOLM: Microscopic microscopic – – and never repeat, and vastly affect the outcome. That’s what?

ELLIE: Unpredictability….

You may be asking, “why am I quoting Jurassic Park in a blog about Amazon”. Great question! And no, it’s not because I just launched a dinosaur themed private label product.


It’s because chaos is all around us, even in our Amazon business.


Think about it. Just like the water drops in Ellie’s hand, we may think our Amazon sales will do one thing, and because of many various reasons, it does another.


How can we be sure what causes your sales stats to change?


What can explain a spike in your sales on a random Tuesday?

  • Maybe A9 bumped you up a few ranks in your money keywords
  • Maybe there was an influx of buyers searching your target keywords
  • Maybe your product is seasonal and that Tuesday was the start of peak season
  • Maybe you made small changes to your listing and now those changes are paying dividends by sending you more traffic.

The Amazon landscape is full of random dips and spikes in conversions and sessions. Sometimes these spikes and dips are volatile, and sometimes they are calm.


So how can we be sure what caused a spike or dip in your sales? Is it because of these random dips/spikes that your sales stats changed, or is it due to the variables you are testing?


In truth, sometimes it is easy to tell the cause of a change in sales stats, and sometimes it is very hard. And as a company that provides you with information on how to optimize your listings through testing, this causes us a dilemma.


So how do we beat chaos?  We utilize statistical significance.


What is Statistical Significance and How We Use it to Make Sure Your Results Are Accurate


Statistical significance is a measurement of how confident we are that the changes you made to your listing (your variables) caused the changes in your sales stats.


It helps you determine if your variables were the actual causes of the changes in your sales stats, or if the changes in your sales stats were due to chaos rearing its ugly head.


If you would like to know more about how we determine statistical significance, check out our last blog.


We express statistical significance in a percent. The higher the percent, the more confident we are.


A good statistical significance is anything above 90%.


This means that we are 90% sure the variable you tested is the cause of any statistical changes, with a 10% chance that random chaos caused these changes.


A 95% statistical significance and above is the best statistical significance you can receive.


A 95% statistical significance almost guarantees that the variable you are testing is the cause of your statistical changes.


We want you to run tests that reach 95% statistical significance and above every time.


How to Run Tests that Reach High Statistical Significance Every Time

  1. Test big ideas and big changes


Generally, a big change to your listing (a bigger variable) will result in bigger changes to your sales statistics.


Bigger changes to your sales statistics will be less likely caused by randomness. Therefore, the bigger the change to your sales stats, the more we are sure that the variables you tested caused the fluctuations in your sales statistics.


For example, say you completely changed you marketing strategy, and decided to write your titles, product features, and descriptions to sell to the opposite sex. You also changed your keywords to reflect this change.


Now, let’s say that your sales dropped 10% shortly after, which was the biggest drop your product ever had. That’s a pretty big decrease.


Because this is such a significant change in your sales statistics right after your change your marketing strategy, we can be more confident that this variable was the cause of this dip in sales.


Another example, is if you changed your lead image and your sessions increase by 25% soon after.


Because there was a big spike in sessions, which occurred soon after you changed that variable, the spike was likely due to that variable.


Although it is true that correlation does not imply causation, we do not just look at the amount of changes in your sales data while your run a test.


We also look at your previous sales data to factor in previous spikes and dips. We also eliminate any anomalies. This way we are not just focusing on correlation.


Here are some big test ideas you can run that would ensure your test receive higher statistical significance:

  1. Changing your content strategy
  2. Changing your lead images
  3. Changing your pricing
  4. Changing your keyword strategy to either keyword stuffing the same keyword or using various keywords throughout your listing


For other ideas, see our blog on testing ideas.

  1. Avoid testing small changes


Now, let’s say you only made a small change to one of the keywords in your product description. Now your sales might not change much at all.


If your sales don’t change much at all, how can we be sure that any change is due to your variable or just to a random change, which is seen everyday?


With a test like this, your statistical significance is likely to be lower because we just can’t be sure.


We try to offset this uncertainty by looking at your previous data and comparing your current changes to that data.


However, the smaller the change the more likely your statistical significance will be smaller.


  1. Continue to drive traffic to your Amazon product page


Don’t change your ad or promotion strategy just because you are running a test. Make sure you continue to drive traffic to your Amazon product page as you were doing before.


Make sure to use Amazon ads, promotions and outside ads in the SAME WAY that you were doing before you started the test. Definitely do not change your ad and promotion strategy during the middle of the test.


Again, DO NOT change your ad and promotion strategy before or during your test, unless you are testing that marketing strategy.


This is because we won’t be sure if changes in your sales data is due to this change in your marketing strategy or the variables you are testing.


For example, let’s say you were driving traffic to your page through Amazon ads. Let’s also say that right before you started to run a test, you decided to stop your ads reducing your sessions to this listing.


Of course, your test results will see a decrease in sessions. and likely sales. But what is the cause of that decrease? The changes to the variables you are testing, or the change to your ad strategy?


The same goes for changing your ad and promotion strategy during testing. Let’s say you stop your ads during the test, decreasing your sessions and sales while one of the variables was running.


Well, that variable would see a decrease in sessions and sales, while the other variables would remain the same.


Again, what is the cause of this decrease? The variable you are testing, or the changes to your ad strategy?

  1. Test one variable at a time:


Splitly allows you to test multiple variables in one test. A variable would be a change in your listing.


For each variable you test, we will run the listing containing that variable for at least 7 days, and then compare it to the other variables in your test. We will also compare this data to your past sales data.


Testing many variables in one test is fine. However, you need to make sure each listing we are going to run only contains one variable change at a time.


Remember, you can run multiple listings, but each listing should only have one change in it.


Think about it this way. Let’s say you are trying to test your money keywords. You have 5  money keywords that you think will drive the most traffic to your listing.


To make this example simple, you only put one money keyword in each listing that you test, so each listing has only one money keyword. So you now have 5 variables you want to test in 5 listings.


Looks good so far. If your sales or sessions increase for one of these listings and the statistical significance comes back high, congratulations, you now know which keyword is your most valuable.


Now, let’s say you decided to test your keywords in the same way as above, but for one of the listings, you lower the sale price of your product.


Now, when the results come back you may find that your sales/sessions jumped for that listing.


You also may find that the statistical significance is high because the data has shown that your changes caused the increase in sales/sessions.


But, which one of your changes caused the increase? Was it your keyword, or was it, more than likely, the decrease in price?


The statistical significance won’t help you determine this, because statistical significance is just telling you how confident we are the changes you made in that listing led to the results you see.


It does not tell you which changes you made caused the results.


Therefore, separate all variables out into separate listings when you are split testing.


What to Do If You Don’t Reach a High Statistical Significance


There are several things you can do to try to increase your statistical significance.


First, try to let the test run longer. The longer you run your tests, the more data we will have.


The more data we have, the more we can determine if your variable is the actual cause or if it was just randomness.


Next, try to make the variables you are testing bigger. As we stated above, the bigger your change, the more likely you are to get bigger results.


The bigger results, the more we can connect it to the variable you changed.




Chaos is all around us, and Amazon sales is no different. Sometimes we can understand why our sales increase or decrease, and sometimes it is just pure randomness.


In order to provide you accurate information on why your sales or sessions are increasing or decreasing, we provide you with statistical significance.


Statistical significance is our way of telling you how confident we are that the variables you are testing caused the changes in your sales or sessions.


There are ways to increase your chances of getting a high statistical significance, including

  • test bigger changes
  • avoid testing small changes
  • continue with your normal ad and promotion strategy
  • test one variable at a time.


If you are getting low statistical significance, try to run your tests for a longer period of time. Also try to make bigger changes to your listings.

Sharing is caring!

Comments 2

  1. Great Post, very helpful! We are running tests on marketing – advertising vs price drop, one variable at a time. How long should we run to get a good “picture” of significance. Should we have a rest time in-between tests or run for same time periods the following week? Holiday shopping is coming up and that could be another variable soon!

    1. Hi Jamie,

      We usually advise you run a split test for at least a week, but even better for two weeks. Splitly will tell you if there has been a statistically significant winner (only variants that have over 90% significance will be declared winners). When it comes to holiday season you are right this may have an impact so it’s a good idea to continually run split tests and make data backed decisions about your listing optimization.

      Here’s a useful post about creating a split testing strategy that you can run throughout the course of a year.

      When it comes to pricing, I would actually advise that you don’t run a split test at all, and instead utilize Profit Peak. This is a tool that utilizes machine learning to continually optimize your price, and this means you stay well ahead of peak seasons, competitor movements, market trends and always have the best known price for optimum sales and rank.

      Hope this helps, please do contact support if you need any help or one-to-one advice on your split testing strategies!


Leave a Reply

Your email address will not be published. Required fields are marked *