Jeremy King has worked in technology for nearly three decades—and has spent much of the past 15 years helping companies use experimentation and data to improve decision-making. Now the senior vice president for technology at Pinterest, King spoke with HBR about the benefits of the experimental approach and the kind of culture that’s necessary to support it. Here are edited excerpts of that conversation:

HBR: When did A/B testing first become part of your work?

King: I worked at eBay from 2001 to 2008, and during the second half of my tenure there experimentation platforms and A/B testing became a focal point. In the early days a lot of experiments focused on the company’s search engine. When I worked at eBay, there were 100 million items for sale on the site, and we were constantly trying to optimize what users were shown when they searched on various terms. The goal was to keep it relevant but also to create serendipity. On platforms such as eBay, Etsy, and Pinterest, you don’t want the search function to be too precise. You want to encourage exploration so that people will roam around and discover new things. So we would experiment with different search results and measure things like transactions, click rates, and exploration time to try to get the best mix.

Does a company require a certain kind of culture to excel at experimentation?

To succeed at it, people have to commit to making decisions based on data. For most established companies, that requires transformational change. In many organizations the senior person in the room, the subject matter expert, or the person with direct responsibility makes the decision unilaterally, often based on instinct. At companies that are data-driven, you are much less likely to hear someone say, “My guess is…” or “I bet that….”

When I joined Pinterest, what impressed me most was that 65% of employees there had done a query in its big-data system in the previous six days. They included not just product engineers and executives but people in HR and on the legal team. In our meetings, if somebody asks a question, instead of guessing at an answer, people’s typical response is to flip open their laptops and begin looking through customer transactions to try to find a data-driven answer.

How hard is it for older, nondigital organizations to shift to that kind of environment?

The number one issue at those firms is that people don’t have access to the data. Organizations like to talk about data democracy, but there are barriers, such as privacy concerns. I get lots of questions from people in all kinds of industries who are skeptical: “Should I really allow the entire company to see all this data?” Data democracy requires an investment and a cultural shift, but the benefits you get from letting more of your company have access to your data are significant, because it unlocks better decision-making.

You spent nearly eight years at Walmart. Describe its experimentation culture.

At Walmart, people still talk about Store Number Eight, which was the location Sam Walton used when he wanted to experiment with some new approach. The practice of selecting a small number of locations where you try out new ideas continues: Walmart has approximately 10 stores it designates for experiments, with at least one in each region. Experiments typically involve things like floor layout or interactive devices. When I was at the company we experimented with a store that had only self-checkout aisles and no cashiers.

As you can imagine, in a physical store experimentation is much slower than it is in a digital environment. Walmart’s culture is also affected by its merchants, who have so much experience that they sometimes rely more on instinct than on data. That instinctual approach can be successful up to a point. But especially when you’re operating at scale and launching thousands of new products each day, as we did at Walmart.com, one person’s ability to understand every new item coming into a category and at what volume it’s going to sell in each region is limited. That task is better left to a computer.