What is AB Testing? – The idea behind AB testing is simple. You have two Variations and you want to find out which one performs better. Imagine the following scenario: your website is currently showing a cat video on your homepage. Suddenly you ask yourself: “Hmm… there are cat people and there are dog people in this world, but which group is bigger?”
Now, your test hypothesis could be something like this: “A dog video probably performs better than a cat video, assuming that there are more dog people on this planet.” Note: you always want to test with a distinct underlying hypothesis, so that you can draw a clear conclusion from your test result.
Your test setup could look like this:
The website traffic to your site is then split equally between both variations and you start measuring your KPIs (Key Performance Indicator). Most of the time the Conversion Rate will be you KPI. The Conversion Rate measures how many people click to the next site or follow the checkout and convert to a paying customer. However, you can always test different KPIs. In the example above it could be video clicks per visitors, to measure the site engagement of your user.
Multi-Variant Testing (MVT)
MVT is often referred to as A/B/C-Testing, this describes best what a MVT is, so instead of having just two candidates A and B, we simply add more candidates C, D, … For example, we could expand our little experiment above to a Variant C which posits that a video showing cats & dogs would perform even better than just one showing either one of them. The test setup could look like this:
As Optimizely pointed out in one of their recent talks: Multi-Variant Testing is key to your success. Optimizely measured the overall success among their Clients and found out that more than two variants lead to a greater overall success. So always try to think of different scenarios to test during your Campaign. However, keep in mind your required sample size. An A/B/C/D Test will usually take longer to reach the required confidence level than a simple AB Test because your traffic is split between multiple candidates. But therefore chances are higher to score a big improvement in one of these candidates.
We hope we could answer the question: what is AB Testing? If you have any suggestions leave a comment below. To find out more about Optimization and on what is AB Testing about? Have a look at our Step by Step Guide.
Become a better marketer through A/B Testing.
Get interesting stuff to your inbox - A/B Test Ideas, proven UX concepts and new tools.