In this webinar session we’re focusing on the fundamental method FBA sellers should be using to improve their sales. We’ll touch on the basic theory behind split testing, plus the best practices so you can use it to increase conversion rates and sales.
As well as invaluable insights from Andy & Ervin, we’ll cover a couple of awesome Amazon Seller Tools we’ve been seeing great success with. So without further ado, let’s get going!
Replay & Slides
As always, here’s your full webinar replay:
Split Testing Is The Best Way To Optimize Amazon Listings For Conversion Rate!
For those who aren’t so familiar with the geeky side of split testing fundamentals, we’re going to break it down to the basic principles. Having a basic understanding of how and why split tests work will enable you to perform them more effectively.
3 Key Benefits of Split Testing
- Increase Conversion Rate (CR) – This is the traditional/primary reason for split testing. Increasing conversion rate helps us FBA sellers sell more units with the same traffic, therefore boosting ROI. But also there’s an extra bonus for us, it pushes us up the Amazon ranking system and gives us more exposure.
- Increase Click-Through Rate (CTR) – If we can optimize our listing to attract more people to click, we increase the amount of traffic and sessions we get from the same impressions. This results in similar benefits, more sales and higher ranking.
- Optimizing Price Point – The final benefit is finding the best price point to balance units sold and profits. Your pricing can affect both your conversion rate and your click-through rate.
For the purpose of this Seller Session, we’re going to focus on optimizing for Conversion Rate, since it’s the primary reason for performing split tests.
Adopting a “Scientific Mindset” For Split Testing
While this may sound a bit over the top, adjusting your mindset is vital to success with split testing. Usually, it’s the new sellers who are most likely to make this mistake but we could all do with a reminder.
Split testing must be logical and data-driven, you must leave your emotions out of it
Don’t Be Unsettled By Peaks & Dips
Sometimes, sellers can get worried that they are going to “break” their listing or lose sales due to changes they make. But that’s not the case, in fact, split testing is the safest way to optimize without that happening!
Also remember, one bad day of sales does not mean that the changes you’ve made are to blame. One bad day of sales could be due to any number of unrelated reasons, even if you hadn’t changed anything.
That’s why split testing is a godsend, it allows us to acquire data over a longer period of time which cancels out any unrelated dips or peaks.
Understanding how this process works will enable you to let go of your emotions and make logical decisions backed up by your own data.
Trust the Mathematics of Probability Behind Split Testing
Split testing is a mathematical process that uses probability to reach relevant conclusions.
The simplest method of explaining the mathematics is to compare this process to the tossing of a fair coin (heads or tails). Let’s say this represents two different versions of your listing.
The diagram shows a coin is tossed and the likelihood of a result. Notice the first result is 50% chance, then the second toss is 50% again, but to get both in sequence drops to 25% chance.
This next diagram gives us an indication of the probability distribution. As we flip the coin 100 times you can see the likelihood of the number of heads is highest at a 50/50 split, then decreasing in probability as you move away.
To achieve the fairest probability distribution, we need to flip the coin more times (or collect more data points).
Relevance to Amazon Split Testing: If you could change your listing back and forth between varients enough times, you will be very likely to get the same conversion rate for both. Unless of course, one version is performing better than the other and that’s what we’ll see in the results.
How We Should Perform Split Testing on Amazon
The conventional way to split test is to switch your listing back and forth every page view. This allows you to collect as many data points (much like a coin toss) as possible. The faster we can switch, the more data points we can gather, however the way Amazon releases data only allows us to do so once every 24 hours.
1. Switching Variants Every 24 Hours
You’ll also need to go to Amazon Seller Central to collect session data for each day you are performing tests. You may need to enter this into an excel spreadsheet so that you can calculate the conversion rate on a daily basis.
2. Collecting Daily Data
3. Analyze the Data
Now you’ve collected enough data, it’s time to start measuring Statistical Significance. This calculation takes the difference and number of data points into account to find the relevance of the data. In other words, it provides you with confidence in the winning variant.
We have a tool online that you can use for this, but there are also many others available online.
Typically we’re looking for at least 90% certainly but 95%+ is better. If you haven’t reached this level of certainly it may be too early to draw a statistically significant conclusion- you’ll need more data points!
As you can see, performing this manually can be quite a beast of a task. That’s why we created our Splitly software suite, to automate this entire reporting and measuring process.
Split Testing on Amazon Question Session
Q. Why do we switch variants every 24 hours, what about 7 Days vs 7 Days?
Actually, that’s what most people do who aren’t fully clued up on split testing.
The problem is, you will have good weeks and bad weeks (just like days). This way of testing could be heavily influenced and skew the results.
With split testing, we should strive for the most coin tosses to even out the data. Every day is the fastest we can gather data from our listing on Amazon.
Q. How does it take to run split tests?
Unfortunately, this is not open an and shut case. There are two factors that come into play:
- Sample Data – more data gives you confidence quicker and this depends on how fast your product is selling
- The Difference – The size of the difference between two variants effects the difference in results. If it was too small, you could keep running forever without seeing much change in result.
That’s why we recommend bigger changes, so you can clearly see the effect within 2 weeks, rather than 1 month or longer!
Q. What are the key factors we should be changing for split testing?
We recommend you start by testing these aspects:
- Main Image
Q. What about testing secondary images?
That’s actually worth trying because on mobile they quite often scroll through images before bullets.
Q. Should we run Splitly’s Profit Peak and AB Testing at the same time?
We don’t allow it because it could cause data corruption. If you start split testing it pauses Profit peak to prevent them from interfering with each others results.
Q. Should we run split testing or Profit Peak first?
Profit peak is simply a case of on/off, allowing it to find the optimal pricing. When it comes to AB testing, it requires more thought on your part to come up with ideas to test.
When you first launch, it’s not a good time to AB test. If you think of any ideas, keep them for later when you’re selling more. Profit Peak is a good place to start because it provides the biggest “bang for your buck”.
Q. Should we avoid split testing older products that already perform well?
Absolutely not. In fact, I’d recommend the opposite.
A lot of sellers think they need to fix the lagging listings and leave the good ones alone. But split testing isn’t good for struggling products because they provide less data.
Think of this another way. How can I make more money? By optimizing your best listings, to sell even more! Testing these products offers the biggest ROI compared to weaker ones.
Just because a product isn’t selling as well doesn’t mean it needs to be optimized. Also, your current best sellers might even have more potential!
You’ll get more data and better answers so they are more suitable for split testing.
Q. Can we still run Lightning Deals or Giveaways during split testing?
Running Lighting Deals will give a boost to one variant or the other, so you would want to dismiss that data. Also, don’t run AB during giveaways. Big changes in sales volume.
The math could get too complicated, or it could skew the results so it’s better to wait.
Q. Can we run promotions over the entire test period, such as coupons FB ads etc?
It should be ok if these effects are evenly distributed across variants. Giveaways can be disruptive. But promotions are usually better even if it ends during testing.
This is why we rotate listings every day, for up to a month, to filter out this kind of noise and disruption.
Q. What’s the best thing we should be doing as FBA sellers to prepare for Q4?
Get ready for the giant sales push.
Get your price point optimized. Or consider reducing the price to increase ranking. Then you could return to optimal pricing. This depends on your niche and competition.
By the time Q4 hits, you want to already have a well-optimized listing, ready to reap profits.
Q. What variables affect how quickly we reach statistical significance?
Overall you should be striving for more data, more sessions, sales and data points.
If you can gather lots of data, plus there’s a big difference between variants, you’ll reach certainty more quickly.
Whereas, if you can’t gather much data, plus there’s only a small difference between variants, certainty will be lower.
Q. If split testing is usually “AB testing”, can we do ABC, ABCD?
What we’re talking about here is multi-variate testing. It means lots of variants will need to be generated to cover all combinations. Since data will be spread across more variants, tests will take far longer to run.
For advanced users with lots of data, Splitly does allow you to do this.
But for the most part, we recommend traditional AB tests. Test each part against each other, get the result quicker and move to the next part. It’s an easier and a more straightforward way to optimize.
Q. What about split testing both the title and main image in one AB test?
If you want to do a whole listing makeover, you could put the new vs old version. It’s a great test to start with. Let’s say new version does better. You take that, then start to drill down with further smaller changes to see.
Q. If you get a winner should you just stop there?
No. Take the winners, keep testing, getting smaller until no significant differences in results can be seen.
If you still have big ideas, keep testing!
More Awesome Amazon Seller Tools To Check Out
Here’s two more Amazon Seller Tools we’ve used that you’ll want to add to your FBA seller arsenal!
Fetcher does an excellent job of keeping Amazon financials in check. We all know what a headache this can be, it’s time-consuming and difficult to figure out. Now imagine kissing goodbye to spreadsheets and excel madness!
Fetcher gets fetches your data, runs calculations and gives you all your finance stats without any fuss. That includes profit, PPC, fees, and many other factors.
JumpSend helps you make more sales, get more reviews and increase organic rankings using promotions and targeted email campaigns.
The ability to create and distribute promotions is great for launching new products. Then the intelligent email automation allows you to reinforce Amazon growth.
Now you have a good understanding of split testing, nothing should stop you from optimizing your listings in time for the sales push. Trust in the data to make informed decisions and leverage all these Amazon Seller Tools to scale your business with ease!