When Designing an A/B Test, Pick Your Metrics Ahead of Time

 

Here is another valuable Management Tip of the Day from Harvard Business Review. To sign up for a free subscription to any/all HBR newsletters, please click here.

*     *     *

A/B testing has become a popular way to evaluate everything from website design to online offers to product descriptions. But many managers don’t design these experiments carefully, relying on software that lets them track hundreds of metrics.

o The problem is that if you’re looking at such a large number of metrics, you’re at risk of seeing what statisticians call spurious correlations. In other words, the more things you’re measuring, the more likely it is that you’ll see random fluctuations.

o You aren’t asking a useful question like “What’s happening with this specific variable?”; you’re asking “What general changes am I seeing?”

o So design a better test by deciding on the metrics you’re going to look at before you execute an experiment. And limit yourself to a few metrics.

If you know exactly what variable you’re interested in and how you’ll measure it, you have a much better chance of seeing (and reporting) significant results.

Adapted from “A Refresher on A/B Testing,” by Amy Gallo

To check out that HBR article and join the discussion, please click here.

Here’s a direct link to dozens of other Management Tips.

 

 

Posted in

Leave a Comment





This site uses Akismet to reduce spam. Learn how your comment data is processed.