BT

Your opinion matters! Please fill in the InfoQ Survey!

A/B Testing at Booking.com

| by João Miranda Follow 2 Followers on Nov 01, 2015. Estimated reading time: 3 minutes |

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

We want customer sentiment to drive product development. Hypothesis proven by experiments are the best to discover customer sentiment. That's how Stuart Frisby, principal designer at booking.com, argued for extensive use of A/B testing to the OSCON attendees in Amsterdam.

A/B testing - the act of comparing different versions of a given feature to understand which one performs better - done right has several prerequisites.

Everything must be tested, but it must be tested atomically. If you don't test one change at a time, you don't control your variables and it becomes impossible to get unambiguous results. Although there are many A/B testing tools on offer, Frisby believes that they are suboptimal, because they lack the context and flexibility needed to do proper and extensive testing. Frisby advocates that you should build your own or at least use tools that you can fix and adapt to your own context.

The organization must build a culture of data-driven product development, instead of relying on the ideas of experts. Hiring people with an entrepreneurial mindset will enable an "ask why" culture that forces everyone to question what it doesn't understand. As the ultimate motivator, good A/B testing will prove many times that you/your boss/the industry experts are wrong. In your context.

Frisby described an hypothetical A/B test, on the effects of changing background colors. This is not an A/B test that Frisby recommends to do in practice, as he believes that changing colors is not the right way to address user problems. But it's simple enough to convey the process. The hypothesis for this experiment would be:

The color of our Primary Call-to-Action [the "Book Now" button - Ed.] is overpowered by the presence of higher contrast elements on the web site.

The metrics that could measure the experiment result:

We'll know if this is true if more people click on a higher contrast button, and end up making a booking.

They would then publish two versions of the button: the one already in use, with a blue background, and a new one with a green background:

Let's assume that the green button led to a decreased conversion rate of 2.2%, down from 2.7%. The hypothesis was not proven, so booking.com would keep the original button.

When starting with A/B testing, the organization must watch out for common mistakes. Don't do "big shot A/B testing", by changing to many things at once. Don't do "fringe A/B testing", by focusing on just a small, even if important, part of your product. Like your landing page, for example. Frisby also dwelled a bit on the idea of "assumed reproducibility".

"Assumed reproducibility" is the idea that experiments made by others can be reproduced in your own setting. But context is king. What may work for others may not work for you. Frisby suggests a hierarchy of reliable data sources (from most to least reliable): your own experiment data; your opinion, because you know your own product best; someone else's opinion; someone else's experiment data, because it provides a false sense of certainty.

Frisby doesn't recommend A/B testing for all scenarios. If your web application does not have enough traffic, your results may not be meaningful. Don't do A/B testing unless you can define objective metrics that enable you to make decisions based on facts. Finally the organization must be prepared to accept that A/B testing will contradict much of what it believes to be right, which is not as easy as it sounds.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT