Seth Roberts writes this morning:
How can you tell when an expert is exaggerating? His lips move.
Some people will misunderstand his point. Roberts is not saying experts exaggerate their conclusions per se. He’s saying experts exaggerate their confidence in their conclusions.
If an expert says that playing a harmonica decreases your risk of influenza by 10%, she’s probably not making that figure up out of thin air (though I am). There probably was some data that implied the 10% figure. It’s not that the data suggested 5% and the scientist said “Let’s call it 10%.” But the quality and quantity of the data may not justify rushing out to buy a harmonica.
One reason experts exaggerate their confidence is that they may be at a loss for words to explain their degree of uncertainty to a popular audience. Journalists can understand “Harmonica playing is good for you” though they probably cannot understand confidence intervals, Bayes factors, or the differences between retrospective versus prospective experiments. (The experts may not really understand these things either.)
Another reason for exaggeration is that you don’t get the attention of the press by making tentative claims. This creates an incentive to suppress uncertainty. But even if experts were transparent regarding their uncertainty, there would still be a selection bias: experts who are sincerely more confident are more likely to be heard.
There are numerous other reasons experts may be wrong, some psychological and some statistical.
I liked the first comment on Roberts’ post:
I tended to rate my colleagues partly by how often the words “I don’t know” passed they lips. Often = good.
In the referenced article, Seth Roberts did not make a good case for the premise that experts exaggerate to any notable extent; what he did was to extrapolate from a single and atypical example that happened to play to his prejudices. I am inclined to believe that there are so-called experts, and even experts with actual expertise, who will exaggerate, but the true experts that I have known have been, to the extent I could tell, impressively in touch with the subtleties and uncertainties of reality. if you have evidence that actual experts exaggerate to an abnormal degree, perhaps it is worth writing about; without it, we just have idle speculation over hearsay.
ARaybould: I believe you can find more examples of expert exaggeration in the book Wrong. Or type “ioannidis” into the search box on this blog.
Agreed; you make a much better argument for the proposition than did Seth Roberts in his post. I have to admit that I have wondered about the level of self-confidence among economists, given their apparent collective (though not in every case) blindsiding, first by the mortgage / CDS crisis and then by the unraveling of the Euro – it is as if climatologists failed to notice global warming until their labs flooded. Perhaps one cause of the problem is when experts form themselves into a self-referential clique?
People often claim that doing xyz will reduce your risk of abc by some% .
But e.g. if the original risk (rarely mentioned) is 10 in 1million and “some” is 20% then 10 in 1 million becomes it becomes 8 in 1 million, which risk is not so very different from 10 in 1 million.
Thus does the media cook up scare stories so as to sell stuff.