When choosing an A/B test winner, there will always be a certain amount of risk.  What risk?  Well, statistically speaking, testing is not a binary proposition with one version a clear loser and one a clear winner.  How then can you be confident in your A/B test results?  Let’s take a look.

I was preparing a lecture for my online course on “CRO Power” (recall that CRO stands for Conversion Rate Optimization), where we have been testing two versions of a pop-up to subscribe to our newsletter in return for an eBook on R.A.C.E. marketing strategy and tactics.  Here are the two versions and you can see which one “won”:

Webdirexion A/B test for popup

Our A/B test results for these two pop-up versions showed that the A version (left) got better results. How significant is that?

Those of you that have read our book on CRO, The Marketer’s Concise Guide to CRO (Amazon print and Kindle versions), know that we like to use a couple of online tools to determine the “significance” of our testing result data.  Here’s a screenshot from Visual Website Optimizer showing our results and significance (a statistical term):

VWO A/B Test significance calsulator

Visual Website Optimizer offers a tool showing the “significance” for your test data.  Test shows about a 44 percent lift for version A.

Now, a logical question would be, how significant are my results?  We can see that the A version performs about 44% better meaning we should get 44 percent more leads — newsletter signups — using the orange version of the popup shown above.  But how confident are we of that?

“…the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages.”

To determine how confident you can be, we can use a “Bayesian” testing calculator.  The name comes from a mathematician named Thomas Bayes (1701–1761), whose statistical theory expresses evidence about the true state of the world in terms of degrees of belief known as Bayesian probabilities. Another concise bit of wisdom to express Mr. Bayes proposition is that nothing is bound in stone.

Ok.  How “bound in stone” are our results?  Fortunately, there is a Bayesian-based testing calculator online courtesy of ABTestGuide.com.  Here’s how our results look in that calculator:

A/B test results in terms of probabilities

The Baysian calculations show A/B test results in terms of probabilities.

Now, we can see that version A (in red on the chart) has 96 percent more of a chance of outperforming version A.  Are we confident, or nervous about the risk we are taking that version B could outperform four percent of the time?  Some cautious CRO experts would recommend we wait longer and gather more data.  In fact, still another online testing tool allows you to set for degree of probability with three choices:  90 percent, 95 percent, and 99 percent.  Ninety-nine percent will be about as close as you can get in the world of statistics.  Oy.

But I chose to end this test, because we have a secret weapon in play using the Bloom pop-up plugin from Elegant Themes.  I go into this more in the course, but the short explanation is that the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages.  Here’s one such example:

Popup A/B Test results in Bloom

The Bloom popup plugin, covered more in my course, CRO Power, offers stats on top converting pages.

The B version shows zero percent conversions on this high converting page, as it does on other such pages. Now I know two things — I have a lot more confidence in the performance of the A version of the pop-up, and I know which pages/posts perform well in terms of conversions — these are potential pages to advertise on Facebook (extend reach and brand awareness) and Google (more for direct sales). We’ll discuss these other actionable aspects in future articles.

We will give your site a Free CRO Review

Send us to your site with a couple of marketing objective notes, and we'll reply with some recommended ways to increase conversions of visitors to leads and customers.

FREE eBook: Subscribe

Subscribe to our Webdirexion R.A.C.E. newsletter and we'll send you a free 34 page eBook on winning your own marketing race using smart tactics (Reach. Act. Convert. Engage.). PLUS... we will enter you in a drawing to win a copy of our book, The Marketer's Concise Guide to CRO.

You have Successfully Subscribed!

Share This

Share this post with your friends!