How can you be confident in your A/B test results? When choosing an A/B test winner, there will always be a certain amount of risk. What risk? Well, statistically speaking, testing is not a binary proposition with one version a clear loser and one a clear winner.
Recently, I was preparing a lecture for my online course on “CRO Power” (recall that CRO stands for Conversion Rate Optimization), where we have been testing two versions of a pop-up to subscribe to our newsletter in return for an eBook on R.A.C.E. marketing strategy and tactics. Here are the two versions and you can see which one “won”:
Those of you that have read our book on CRO, The Marketer’s Concise Guide to CRO (Amazon print and Kindle versions), know that we like to use a couple of online tools to determine the “significance” of our testing result data. Here’s a screenshot from Visual Website Optimizer showing our results and significance (a statistical term):
Now, a logical question would be, how significant are my results? We can see that the A version performs about 44% better meaning we should get 44 percent more leads — newsletter signups — using the orange version of the popup shown above. But how confident are we of that?
“…the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages.”
To determine how confident you can be, we can use a “Bayesian” testing calculator. The name comes from a mathematician named Thomas Bayes (1701–1761), whose statistical theory expresses evidence about the true state of the world in terms of degrees of belief known as Bayesian probabilities. Another concise bit of wisdom to express Mr. Bayes proposition is that nothing is bound in stone.
Ok. How “bound in stone” are our results? Fortunately, there is a Bayesian-based testing calculator online courtesy of ABTestGuide.com. Here’s how our results look in that calculator:
Now, we can see that version A (in red on the chart) has 96 percent more of a chance of outperforming version A. Are we confident, or nervous about the risk we are taking that version B could outperform four percent of the time? Some cautious CRO experts would recommend we wait longer and gather more data. In fact, still another online testing tool allows you to set for degree of probability with three choices: 90 percent, 95 percent, and 99 percent. Ninety-nine percent will be about as close as you can get in the world of statistics. Oy.
But I chose to end this test, because we have a secret weapon in play using the Bloom pop-up plugin from Elegant Themes. I go into this more in the course, but the short explanation is that the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages. Here’s one such example:
The B version shows zero percent conversions on this high converting page, as it does on other such pages. Now I know two things — I have a lot more confidence in the performance of the A version of the pop-up, and I know which pages/posts perform well in terms of conversions — these are potential pages to advertise on Facebook (extend reach and brand awareness) and Google (more for direct sales). We’ll discuss these other actionable aspects in future articles.