Here is an excerpt from an article written by Malcolm Gladwell for The New Yorker, dated June 24th, 2013 and included among the articles accessible in the magazine’s archive. To read all of this article and check out others, please visit Malcolm’s website by clicking here.

* * *

In the mid-nineteenth century, work began on a crucial section of the railway line connecting Boston to the Hudson River. The addition would run from Greenfield, Massachusetts, to Troy, New York, and it required tunnelling through Hoosac Mountain, a massive impediment, nearly five miles thick, that blocked passage between the Deerfield Valley and a tributary of the Hudson.

James Hayward, one of New England’s leading railroad engineers, estimated that penetrating the Hoosac would cost, at most, a very manageable two million dollars. The president of Amherst College, an accomplished geologist, said that the mountain was composed of soft rock and that tunnelling would be fairly easy once the engineers had breached the surface. “The Hoosac . . . is believed to be the only barrier between Boston and the Pacific,” the project’s promoter, Alvah Crocker, declared.

Everyone was wrong. Digging through the Hoosac turned out to be a nightmare. The project cost more than ten times the budgeted estimate. If the people involved had known the true nature of the challenges they faced, they would never have funded the Troy-Greenfield railroad. But, had they not, the factories of northwestern Massachusetts wouldn’t have been able to ship their goods so easily to the expanding West, the cost of freight would have remained stubbornly high, and the state of Massachusetts would have been immeasurably poorer. So is ignorance an impediment to progress or a precondition for it?

The economist Albert O. Hirschman (above), who died last December, loved paradoxes like this. He was a “planner,” the kind of economist who conceives of grand infrastructure projects and bold schemes. But his eye was drawn to the many ways in which plans did not turn out the way they were supposed to—to unintended consequences and perverse outcomes and the puzzling fact that the shortest line between two points is often a dead end.

“The Principle of the Hiding Hand,” one of Hirschman’s many memorable essays, drew on an account of the Troy-Greenfield “folly,” and then presented an even more elaborate series of paradoxes. Hirschman had studied the enormous Karnaphuli Paper Mills, in what was then East Pakistan. The mill was built to exploit the vast bamboo forests of the Chittagong Hill Tracts. But not long after the mill came online the bamboo unexpectedly flowered and then died, a phenomenon now known to recur every fifty years or so. Dead bamboo was useless for pulping; it fell apart as it was floated down the river. Because of ignorance and bad planning, a new, multimillion-dollar industrial plant was suddenly without the raw material it needed to function.

But what impressed Hirschman was the response to the crisis. The mill’s operators quickly found ways to bring in bamboo from villages throughout East Pakistan, building a new supply chain using the country’s many waterways. They started a research program to find faster-growing species of bamboo to replace the dead forests, and planted an experimental tract. They found other kinds of lumber that worked just as well. The result was that the plant was blessed with a far more diversified base of raw materials than had ever been imagined. If bad planning hadn’t led to the crisis at the Karnaphuli plant, the mill’s operators would never have been forced to be creative. And the plant would not have been nearly as valuable as it became.

“We may be dealing here with a general principle of action,” Hirschman wrote:

Creativity always comes as a surprise to us; therefore we can never count on it and we dare not believe in it until it has happened. In other words, we would not consciously engage upon tasks whose success clearly requires that creativity be forthcoming. Hence, the only way in which we can bring our creative resources fully into play is by misjudging the nature of the task, by presenting it to ourselves as more routine, simple, undemanding of genuine creativity than it will turn out to be.

And from there Hirschman’s analysis took flight. People don’t seek out challenges, he went on. They are “apt to take on and plunge into new tasks because of the erroneously presumed absence of a challenge—because the task looks easier and more manageable than it will turn out to be.” This was the Hiding Hand principle—a play on Adam Smith’s Invisible Hand. The entrepreneur takes risks but does not see himself as a risk-taker, because he operates under the useful delusion that what he’s attempting is not risky. Then, trapped in mid-mountain, people discover the truth—and, because it is too late to turn back, they’re forced to finish the job.

* * *

Here is a direct link to the complete article.

Malcolm Gladwell has been a staff writer at The New Yorker since 1996. He is the author of The Tipping Point, Blink, Outliers, What the Dog Saw, and most recently, David and Goliath. Prior to joining The New Yorker, he was a reporter at the Washington Post. Gladwell was born in England and grew up in rural Ontario. He now lives in New York.