Here is a brief excerpt from an article by Michael Schrage, featured in the MIT Sloan Management Review. Today’s savviest executives acknowledge that most of the time, they can’t accurately predict what their customers will want, like or need. That’s where experiments come in. Too many executives worldwide would rather use their expertise to intuit what’s best for their customers than run even the simplest experiment. They take pride in their “gut.” That’s a huge managerial mistake. This new article from the winter issue describes how companies such as Quicken and Microsoft are using experimentation to determine what their customers want.

To read the full article, check out others, obtain subscription information, and register for free access to some resources, please click here.

* * *

Microsoft’s Ronny Kohavi, a pioneer in online experimentation since his days as director of data mining and personalization at Amazon.com, likes to challenge his Internet-savvy audiences with a little experiment. Kohavi, a Microsoft distinguished engineer and general manager for analysis and experimentation, shows screenshots of simple but real A/B experiments that Microsoft ran to test different Web design options. Some visitors saw the A version, some the B version — and Microsoft tracked which version yielded better results. Kohavi then challenges his audience to predict the real-world outcomes of the test: Did Option A win? Did Option B win? Or were the results a statistical dead heat?

Almost without exception, his audiences fragment; not even a rough consensus emerges. “Stop debating,” Kohavi advises. “It’s easier to get the data…Data trumps intuition.”

Data from experiments such as Kohavi’s should ideally encourage executive curiosity and humility. Unfortunately, that rarely happens. Too many executives worldwide would rather use their expertise to “think” or “intuit” what’s best for their customers than run even the simplest experiment. They take pride in their “gut.” That’s a huge managerial mistake.

Organizations may be confident they know their customers, but they’re very likely to be overconfident. Most executives aren’t nearly as smart, perceptive or customer-centric as they believe. Leading researchers, such as Daniel Kahneman, who received the 2002 Nobel Prize in Economic Sciences, and psychologist Philip E. Tetlock, have persuasively demonstrated that “experts” are particularly prone to predictive overconfidence.

For example, in a study where experts were asked to make predictions related to world events, Tetlock reported in a 1999 article that “across all seven scenarios, experts were only slightly more accurate than one would expect from chance.” What’s more, Tetlock found that the “experts who assigned confidence estimates of 80 percent or higher were correct only 45 percent of the time.” The overconfidence — arrogance? — of presumed expertise is counterproductive. Pride goeth before a fall. Data trumps intuition. Serious innovators take data seriously.

Michael Schrage is a research fellow at the MIT Center for Digital Business and the author of The Innovator’s Hypothesis: How Cheap Experiments Are Worth More Than Good Ideas (MIT Press, 2014). To read my review of it, please click here.