Our A/B test results for these two pop-up versions showed that the A version (left) got better results. How significant is that?
Those of you that have read our book on CRO, The Marketer’s Concise Guide to CRO (Amazon print and Kindle versions), know that we like to use a couple of online tools to determine the “significance” of our testing result data. Here’s a screenshot from Visual Website Optimizer showing our results and significance (a statistical term):
Visual Website Optimizer offers a tool showing the “significance” for your test data. Test shows about a 44 percent lift for version A.
Now, a logical question would be, how significant are my results? We can see that the A version performs about 44% better meaning we should get 44 percent more leads — newsletter signups — using the orange version of the popup shown above. But how confident are we of that?
“…the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages.”
To determine how confident you can be, we can use a “Bayesian” testing calculator. The name comes from a mathematician named Thomas Bayes (1701–1761), whose statistical theory expresses evidence about the true state of the world in terms of degrees of belief known as Bayesian probabilities. Another concise bit of wisdom to express Mr. Bayes proposition is that nothing is bound in stone.
Ok. How “bound in stone” are our results? Fortunately, there is a Bayesian-based testing calculator online courtesy of ABTestGuide.com. Here’s how our results look in that calculator:
The Baysian calculations show A/B test results in terms of probabilities.
Now, we can see that version A (in red on the chart) has 96 percent more of a chance of outperforming version A. Are we confident, or nervous about the risk we are taking that version B could outperform four percent of the time? Some cautious CRO experts would recommend we wait longer and gather more data. In fact, still another online testing tool allows you to set for degree of probability with three choices: 90 percent, 95 percent, and 99 percent. Ninety-nine percent will be about as close as you can get in the world of statistics. Oy.
But I chose to end this test, because we have a secret weapon in play using the Bloom pop-up plugin from Elegant Themes. I go into this more in the course, but the short explanation is that the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages. Here’s one such example:
The Bloom popup plugin, covered more in my course, CRO Power, offers stats on top converting pages.
The B version shows zero percent conversions on this high converting page, as it does on other such pages. Now I know two things — I have a lot more confidence in the performance of the A version of the pop-up, and I know which pages/posts perform well in terms of conversions — these are potential pages to advertise on Facebook (extend reach and brand awareness) and Google (more for direct sales). We’ll discuss these other actionable aspects in future articles.
