How accurate are media experts?

[BWV 1080]

Every day, experts bombard us with their views on topics as varied as Iraqi insurgents, Bolivian coca growers, European central bankers, and North Korea’s Politburo. But how much credibility should we attach to the opinions of experts?

The sanguine view is that as long as those selling expertise compete vigorously for the attention of discriminating buyers (the mass media), market mechanisms will assure quality control. Pundits who make it into newspaper opinion pages or onto television and radio must have good track records; otherwise, they would have been weeded out.

Skeptics, however, warn that the mass media dictate the voices we hear and are less interested in reasoned debate than in catering to popular prejudices. As a result, fame could be negatively, not positively, correlated with long-run accuracy.

Until recently, no one knew who is right, because no one was keeping score. But the results of a 20-year research project now suggest that the skeptics are closer to the truth.

I describe the project in detail in my book Expert Political Judgment: How good is it? How can we know? The basic idea was to solicit thousands of predictions from hundreds of experts about the fates of dozens of countries, and then score the predictions for accuracy. We find that the media not only fail to weed out bad ideas, but that they often favor bad ideas, especially when the truth is too messy to be packaged neatly.

The evidence falls into two categories. First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers.

Between 1985 and 2005, boomsters made 10-year forecasts that exaggerated the chances of big positive changes in both financial markets (e.g., a Dow Jones Industrial Average of 36,000) and world politics (e.g., tranquility in the Middle East and dynamic growth in sub-Saharan Africa). They assigned probabilities of 65% to rosy scenarios that materialized only 15% of the time.

In the same period, doomsters performed even more poorly, exaggerating the chances of negative changes in all the same places where boomsters accentuated the positive, plus several more (I still await the impending disintegration of Canada, Nigeria, India, Indonesia, South Africa, Belgium, and Sudan). They assigned probabilities of 70% to bleak scenarios that materialized only 12% of the time.

Second, again as the skeptics warned, over-claimers rarely pay penalties for being wrong. Indeed, the media shower lavish attention on over-claimers while neglecting their humbler colleagues.

We can see this process in sharp relief when, following the philosopher Sir Isaiah Berlin, we classify experts as “hedgehogs” or “foxes.” Hedgehogs are big-idea thinkers in love with grand theories: libertarianism, Marxism, environmentalism, etc. Their self-confidence can be infectious. They know how to stoke momentum in an argument by multiplying reasons why they are right and others are wrong.

That wins them media acclaim. But they don’t know when to slam the mental brakes by making concessions to other points of view. They take their theories too seriously. The result: hedgehogs make more mistakes, but they pile up more hits on Google.

Eclectic foxes are better at curbing their ideological enthusiasms. They are comfortable with protracted uncertainty about who is right even in bitter debates, conceding gaps in their knowledge and granting legitimacy to opposing views. They sprinkle their conversations with linguistic qualifiers that limit the reach of their arguments: ‘but,’ ‘however,’ ‘although.’

Because they avoid over-simplification, foxes make fewer mistakes. Foxes will often agree with hedgehogs up to a point, before complicating things: “Yes, my colleague is right that the Saudi monarchy is vulnerable, but remember that coups are rare and that the government commands many means of squelching opposition.”

Imagine your job as a media executive depends on expanding your viewing audience. Whom would you pick: an expert who balances conflicting arguments and concludes that the likeliest outcome is more of the same, or an expert who gets viewers on the edge of their seats over radical Islamists seizing control and causing oil prices to soar?

In short, the qualities that make foxes more accurate also make them less popular.

At this point, uncharitable skeptics chortle that we get the media we deserve. But that is unfair. No society has yet created a widely trusted method for keeping score on the punditocracy. Even citizens who prize accuracy have little way of knowing that they are sacrificing it when they switch channels from boring foxes to charismatic hedgehogs.

Here, then, is a modest proposal that applies to all democracies: the marketplace of ideas works better if it is easier for citizens to see the trade-offs between accuracy and entertainment, or between accuracy and party loyalty. Wouldn’t they be more likely to read pundits with better track records? If so, pundits might adapt to accountability by showing more humility, and political debate might begin to sound less shrill.

Granted, it is not easy to create methods of keeping score that are credible across the spectrum of reasonable opinion. But in a world where, as Yeats said, “and the best lack all conviction, while the worst are full of passionate intensity,” it is worth trying.

[Corlyss_D]

How accurate are media experts?

I'd rather use a .40 H&K USP Compact myself.

[Ralph]

How accurate are media experts?

I'd rather use a .40 H&K USP Compact myself.

*****

Meaning?

[Corlyss_D]

How accurate are media experts?

I'd rather use a .40 H&K USP Compact myself.

*****

Meaning?

Play on "accuracy." The media are sooooooo unreliable, if they were the gun they fancy themselves to be, you'd throw them away.

[Ralph]

How accurate are media experts?

I'd rather use a .40 H&K USP Compact myself.

*****

Meaning?

Play on "accuracy." The media are sooooooo unreliable, if they were the gun they fancy themselves to be, you'd throw them away.

*****

Oh. Deep.