How to Know the Difference Between Your Data and Your Metrics

How to KnowHere is an excerpt from an article written by Jeff Bladt and Bob Filbin for Harvard Business Review and the HBR Blog Network. To read the complete article, check out the wealth of free resources, and sign up for a subscription to HBR email alerts, please click here.

* * *

How many views make a YouTube video a success? How about 1.5 million? That’s how many views a video our organization, DoSomething.org, posted in 2011 got. It featured some well-known YouTube celebrities, who asked young people to donate their used sports equipment to youth in need. It was twice as popular as any video Dosomething.org had posted to date. Success! Then came the data report: only eight viewers had signed up to donate equipment, and zero actually donated.

Zero donations. From 1.5 million views. Suddenly, it was clear that for DoSomething.org, views did not equal success. In terms of donations, the video was a complete failure.

What happened? We were concerned with the wrong metric. A metric contains a single type of data, e.g., video views or equipment donations. A successful organization can only measure so many things well and what it measures ties to its definition of success. For DoSomething.org, that’s social change. In the case above, success meant donations, not video views. As we learned, there is a difference between numbers and numbers that matter. This is what separates data from metrics.

You can’t pick your data, but you must pick your metrics.

Take baseball. Every team has the same definition of success — winning the World Series. This requires one main asset: good players. But what makes a player good? In baseball, teams used to answer this question with a handful of simple metrics like batting average and runs batted in (RBIs). Then came the statisticians (remember Moneyball?). New metrics provided teams with the ability to slice their data in new ways, find better ways of defining good players, and thus win more games.

Keep in mind that all metrics are proxies for what ultimately matters (in the case of baseball, a combination of championships and profitability), but some are better than others. The data of the game has never changed — there are still RBIs and batting averages; what has changed is how we look at the data. And those teams that slice the data in smarter ways are able to find good players that have been traditionally undervalued.

* * *

Organizations can’t control their data, but they do control what they care about. If our metric on the YouTube video had been views, we would have called it a huge success. In fact, we wrote it off as a massive failure. Does that mean no more videos? Not necessarily, but for now, we’ll be spending our resources elsewhere, collecting data on metrics that matter. Good data scientists know that analyzing the data is the easy part. The hard part is deciding what data matters.

* * *

To read the complete article, please click here.

Bladt & FilbinJeff Bladt is Director of Data Products & Analytics at DoSomething.org, America’s largest organization for teens and social change. Bob Filbin is Chief Data Scientist at Crisis Text Line , the first large-scale 24/7 national crisis line for teens on the medium they use most: texting. The service will launch August 1st.

Posted in

Leave a Comment





This site uses Akismet to reduce spam. Learn how your comment data is processed.