Businesses started caring a lot more about diversity after a series of high-profile lawsuits rocked the financial industry. In the late 1990s and early 2000s, Morgan Stanley shelled out $54 million—and Smith Barney and Merrill Lynch more than $100 million each—to settle sex discrimination claims. In 2007, Morgan was back at the table, facing a new class action, which cost the company $46 million. In 2013, Bank of America Merrill Lynch settled a race discrimination suit for $160 million. Cases like these brought Merrill’s total 15-year payout to nearly half a billion dollars.

It’s no wonder that Wall Street firms now require new hires to sign arbitration contracts agreeing not to join class actions. They have also expanded training and other diversity programs. But on balance, equality isn’t improving in financial services or elsewhere. Although the proportion of managers at U.S. commercial banks who were Hispanic rose from 4.7% in 2003 to 5.7% in 2014, white women’s representation dropped from 39% to 35%, and black men’s from 2.5% to 2.3%. The numbers were even worse in investment banks (though that industry is shrinking, which complicates the analysis). Among all U.S. companies with 100 or more employees, the proportion of black men in management increased just slightly—from 3% to 3.3%—from 1985 to 2014. White women saw bigger gains from 1985 to 2000—rising from 22% to 29% of managers—but their numbers haven’t budged since then. Even in Silicon Valley, where many leaders tout the need to increase diversity for both business and social justice reasons, bread-and-butter tech jobs remain dominated by white men.

It shouldn’t be surprising that most diversity programs aren’t increasing diversity. Despite a few new bells and whistles, courtesy of big data, companies are basically doubling down on the same approaches they’ve used since the 1960s—which often make things worse, not better. Firms have long relied on diversity training to reduce bias on the job, hiring tests and performance ratings to limit it in recruitment and promotions, and grievance systems to give employees a way to challenge managers. Those tools are designed to preempt lawsuits by policing managers’ thoughts and actions. Yet laboratory studies show that this kind of force-feeding can activate bias rather than stamp it out. As social scientists have found, people often rebel against rules to assert their autonomy. Try to coerce me to do X, Y, or Z, and I’ll do the opposite just to prove that I’m my own person.In analyzing three decades’ worth of data from more than 800 U.S. firms and interviewing hundreds of line managers and executives at length, we’ve seen that companies get better results when they ease up on the control tactics. It’s more effective to engage managers in solving the problem, increase their on-the-job contact with female and minority workers, and promote social accountability—the desire to look fair-minded. That’s why interventions such as targeted college recruitment, mentoring programs, self-managed teams, and task forces have boosted diversity in businesses. Some of the most effective solutions aren’t even designed with diversity in mind.

Here, we dig into the data, the interviews, and company examples to shed light on what doesn’t work and what does.

Why You Can’t Just Outlaw Bias

Executives favor a classic command-and-control approach to diversity because it boils expected behaviors down to dos and don’ts that are easy to understand and defend. Yet this approach also flies in the face of nearly everything we know about how to motivate people to make changes. Decades of social science research point to a simple truth: You won’t get managers on board by blaming and shaming them with rules and reeducation. Let’s look at how the most common top-down efforts typically go wrong.

Diversity training.

Do people who undergo training usually shed their biases? Researchers have been examining that question since before World War II, in nearly a thousand studies. It turns out that while people are easily taught to respond correctly to a questionnaire about bias, they soon forget the right answers. The positive effects of diversity training rarely last beyond a day or two, and a number of studies suggest that it can activate bias or spark a backlash. Nonetheless, nearly half of midsize companies use it, as do nearly all the Fortune 500.

Many firms see adverse effects. One reason is that three-quarters use negative messages in their training. By headlining the legal case for diversity and trotting out stories of huge settlements, they issue an implied threat: “Discriminate, and the company will pay the price.” We understand the temptation—that’s how we got your attention in the first paragraph—but threats, or “negative incentives,” don’t win converts.

Another reason is that about three-quarters of firms with training still follow the dated advice of the late diversity guru R. Roosevelt Thomas Jr. “If diversity management is strategic to the organization,” he used to say, diversity training must be mandatory, and management has to make it clear that “if you can’t deal with that, then we have to ask you to leave.” But five years after instituting required training for managers, companies saw no improvement in the proportion of white women, black men, and Hispanics in management, and the share of black women actually decreased by 9%, on average, while the ranks of Asian-American men and women shrank by 4% to 5%. Trainers tell us that people often respond to compulsory courses with anger and resistance—and many participants actually report more animosity toward other groups afterward.

But voluntary training evokes the opposite response (“I chose to show up, so I must be pro-diversity”), leading to better results: increases of 9% to 13% in black men, Hispanic men, and Asian-American men and women in management five years out (with no decline in white or black women). Research from the University of Toronto reinforces our findings: In one study white subjects read a brochure critiquing prejudice toward blacks. When people felt pressure to agree with it, the reading strengthened their bias against blacks. When they felt the choice was theirs, the reading reduced bias.

Companies too often signal that training is remedial. The diversity manager at a national beverage company told us that the top brass uses it to deal with problem groups. “If there are a number of complaints…or, God forbid, some type of harassment case…leaders say, ‘Everyone in the business unit will go through it again.’” Most companies with training have special programs for managers. To be sure, they’re a high-risk group because they make the hiring, promotion, and pay decisions. But singling them out implies that they’re the worst culprits. Managers tend to resent that implication and resist the message.