We wish to express our gratitude to Jonah Lehrer for his permission to share an excerpt from Chapter 7, The Brain Is An Argument, in his 2009 bestseller “How We Decide“. To fully understand the context of the excerpt I recommend that you read the book.
Mr. Lehrer wrote, “There’s something unsettling about seeing the brain as one big argument. We might like to believe that our decisions reflect a clear cortical consensus, that the entire mind agrees on what we should do. And yet, that serene self-image has little basis in reality.”
Having spent the last ten years in an industry filled with pundits serving clients who have equally large egos and preconceptions, Lehrer’s coverage of Philip Tetlock’s research results struck a chord with me. I hope to revisit this topic in a future post and discuss how it affects decision-making and our willingness to consider/accept the opposing views and opinions of others.
In 1984, the University of California at Berkeley psychologist Philip Tetlock began what he thought would be a brief research project. At the time, the Cold War was flaring up again—Reagan was talking tough to the ‘evil empire’—and political pundits were sharply divided on the wisdom of American foreign policy. The doves thought that Reagan was needlessly antagonizing the Soviets, while the hawks where convinced that the USSR needed to be aggressively contained. Tetlock was curious which group of pundits would turn out to be right, and so he began monitoring their predictions.
A few years later, after Reagan left office, Tetlock revisited the opinions of the pundits. His conclusion was sobering: everyone was wrong. The doves had assumed that Reagan’s bellicose stance would exacerbate Cold War tensions and had predicted a breakdown in diplomacy as the USSR hardened its geopolitical stance. The reality, of course, was that the exact opposite happened. By 1985, Mikhail Gorbachev was in power. The Soviet Union began implementing a stunning series of internal reforms. The ‘evil empire’ was undergoing glasnost.
But the hawks didn’t do much better. Even after Gorbachev began the liberalizing process, hawks tended to disparage the changes to the Soviet system. they said the evil empire was still evil; Gorbachev was just a tool of the politburo. Hawks couldn’t imagine that a sincere reformer might actually emerge from a totalitarian state.
The dismal performance of these pundits inspired Tetlock to turn his small case study into an epic experimental project. He picked 284 people who made their living ‘commenting or offering advice on political and economic trends’ and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be reelected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case the pundits were ask to rate the probability of several possible outcomes. Tetlock then interrogated the pundits about their thought processes so he could better understand how they’d made up their minds. By the end of the study, Tetlock had quantified 82,361 different predictions.
After Tetlock tallied the data, the predictive failures of the pundits became obvious. Although they were paid for their keen insights in world affairs, they tended to perform worse than random chance. Most of Tetlock’s questions had three possible answers; on average, the pundits had selected the right answer less than 33 percent of the time. In other words, a dart-throwing chimp would have beaten the vast majority of professionals. Tetlock also found that the most famous pundits in his study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.
Why were these pundits (especially the prominent ones) so bad at forecasting the future? The central error diagnosed by Tetlock was the sin of certainty, which led the ‘experts’ to mistakenly impose a top-down solution on their decision making processes. In chapter 2, we saw examples of the true expertise that occurs when experience is internalized by the dopamine system. This results in a person who has a set of instincts that respond quickly to the situation at hand, regardless of whether that’s playing backgammon or staring at a radar screen. The pundits in Tetlock’s study, however, distorted the verdicts of their emotional brains, cherry-picking the feelings they wanted to follow. Instead of trusting their gut feelings, they found ways to disregard the insights that contradicted their ideologies. When pundits were convinced they were right, they ignored any brain areas that implied they might be wrong. This suggests that one of the best ways to distinguish genuine from phony expertise is to look at how a person responds to dissonant data. Does he or she reject the data out of hand? Perform elaborate mental gymnastics to avoid admitting error? Everyone makes mistakes; the object is to learn from these mistakes.
Tetlock notes that the best pundits are willing to state their opinions in ‘testable form’ so they can ‘continually monitor their forecasting performance.’ He argues that this approach makes pundits not only more responsible—they are forced to account for being wrong—but also less prone to bombastic convictions, a crucial sign that the pundit isn’t worth listening to. (In other words, ignore those commentators that seem too confident or self-assured. The people on television who are most certain are almost certainly going to be wrong.) As Tetlock writes, ‘The dominant danger [for pundits] remains hubris, the vice of closed-mindedness, of dismissing dissonant possibilities too quickly.’Even though practically all professionals in Tetlock’s study claimed that they were dispassionately analyzing the evidence—everybody wanted to be rational—many of them were actually indulging in some conveniently cultivated ignorance. Instead of encouraging the arguments inside their heads, these pundits settled on answers and then came up with reasons to justify those answers. They were, as Tetlock put it, ‘prisoners of their preconceptions.’