Those of you that have read our book on CRO, The Marketer’s Concise Guide to CRO (Amazon print and Kindle versions), know that we like to use a couple of online tools to determine the “significance” of our testing result data. Here’s a screenshot from Visual Website Optimizer showing our results and significance (a statistical term):
Now, a logical question would be, how significant are my results? We can see that the A version performs about 44% better meaning we should get 44 percent more leads — newsletter signups — using the orange version of the popup shown above. But how confident are we of that?
“…the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages.”
To determine how confident you can be, we can use a “Bayesian” testing calculator. The name comes from a mathematician named Thomas Bayes (1701–1761), whose statistical theory expresses evidence about the true state of the world in terms of degrees of belief known as Bayesian probabilities. Another concise bit of wisdom to express Mr. Bayes proposition is that nothing is bound in stone.
Ok. How “bound in stone” are our results? Fortunately, there is a Bayesian-based testing calculator online courtesy of ABTestGuide.com. Here’s how our results look in that calculator:
Now, we can see that version A (in red on the chart) has 96 percent more of a chance of outperforming version A. Are we confident, or nervous about the risk we are taking that version B could outperform four percent of the time? Some cautious CRO experts would recommend we wait longer and gather more data. In fact, still another online testing tool allows you to set for degree of probability with three choices: 90 percent, 95 percent, and 99 percent. Ninety-nine percent will be about as close as you can get in the world of statistics. Oy.
But I chose to end this test, because we have a secret weapon in play using the Bloom pop-up plugin from Elegant Themes. I go into this more in the course, but the short explanation is that the Bloom plugin not only does a simple calculation on results, but it then shows you specific a/b test results on your highest performing pages. Here’s one such example:
The B version shows zero percent conversions on this high converting page, as it does on other such pages. Now I know two things — I have a lot more confidence in the performance of the A version of the pop-up, and I know which pages/posts perform well in terms of conversions — these are potential pages to advertise on Facebook (extend reach and brand awareness) and Google (more for direct sales). We’ll discuss these other actionable aspects in future articles.
We will give your site a Free CRO ReviewSend us to your site with a couple of marketing objective notes, and we'll reply with some recommended ways to increase conversions of visitors to leads and customers.
Scott serves as CMO & Chief Optimizer on the Webdirexion LLC agency digital marketing team for both development and content marketing strategy, and is the author of the new book, “The Marketer’s Concise Guide to CRO” (Oct. 2015). He is a career marketing communications professional with niche industry specialties in healthcare, law firms, and hotel marketing and holds recent certifications in Google Analytics Mastery (Udemy), and RACE Digital Marketing (Smart Insights). He is currently working toward Professional Certified Marketer (PCM) in Digital Marketing through the American Marketing Association which also requires annual training and 3-year recertification.
Scott has also taught business, web programming and eCommerce courses at colleges in the Portland, Oregon area. He currently teaches WordPress Content Marketing Power, an online course through Udemy, and has spoken at several Content Marketing conferences. When he is not geeking out on a Mac, Nexus 7, or Google Chromebook, he enjoys Tai Chi, walking with his dog (Steggman), and camping trips. Scott says he survives on Coffee and Pizza.