Occasionally, a new way of organizing work leads to extraordinary improvements. Total quality management did that in manufacturing in the 1980s by combining a set of tools—kanban cards, quality circles, and so on—with the insight that people on the shop floor could do much higher level work than they usually were asked to. That blend of tools and insight, applied to a work process, can be thought of as a social technology.

In a recent seven-year study in which I looked in depth at 50 projects from a range of sectors, including business, health care, and social services, I have seen that another social technology, design thinking, has the potential to do for innovation exactly what TQM did for manufacturing: unleash people’s full creative energies, win their commitment, and radically improve processes. By now most executives have at least heard about design thinking’s tools—ethnographic research, an emphasis on reframing problems and experimentation, the use of diverse teams, and so on—if not tried them. But what people may not understand is the subtler way that design thinking gets around the human biases (for example, rootedness in the status quo) or attachments to specific behavioral norms (“That’s how we do things here”) that time and again block the exercise of imagination.

In this article I’ll explore a variety of human tendencies that get in the way of innovation and describe how design thinking’s tools and clear process steps help teams break free of them. Let’s begin by looking at what organizations need from innovation—and at why their efforts to obtain it often fall short.

The Challenges of Innovation

To be successful, an innovation process must deliver three things: superior solutions, lower risks and costs of change, and employee buy-in. Over the years businesspeople have developed useful tactics for achieving those outcomes. But when trying to apply them, organizations frequently encounter new obstacles and trade-offs.

Superior solutions.

Defining problems in obvious, conventional ways, not surprisingly, often leads to obvious, conventional solutions. Asking a more interesting question can help teams discover more-original ideas. The risk is that some teams may get indefinitely hung up exploring a problem, while action-oriented managers may be too impatient to take the time to figure out what question they should be asking.

It’s also widely accepted that solutions are much better when they incorporate user-driven criteria. Market research can help companies understand those criteria, but the hurdle here is that it’s hard for customers to know they want something that doesn’t yet exist.

Finally, bringing diverse voices into the process is also known to improve solutions. This can be difficult to manage, however, if conversations among people with opposing views deteriorate into divisive debates.

Lower risks and costs.

Uncertainty is unavoidable in innovation. That’s why innovators often build a portfolio of options. The trade-off is that too many ideas dilute focus and resources. To manage this tension, innovators must be willing to let go of bad ideas—to “call the baby ugly,” as a manager in one of my studies described it. Unfortunately, people often find it easier to kill the creative (and arguably riskier) ideas than to kill the incremental ones.

Employee buy-in.

An innovation won’t succeed unless a company’s employees get behind it. The surest route to winning their support is to involve them in the process of generating ideas. The danger is that the involvement of many people with different perspectives will create chaos and incoherence.

Underlying the trade-offs associated with achieving these outcomes is a more fundamental tension. In a stable environment, efficiency is achieved by driving variation out of the organization. But in an unstable world, variation becomes the organization’s friend, because it opens new paths to success. However, who can blame leaders who must meet quarterly targets for doubling down on efficiency, rationality, and centralized control?

To manage all the trade-offs, organizations need a social technology that addresses these behavioral obstacles as well as the counterproductive biases of human beings. And as I’ll explain next, design thinking fits that bill.