In his autobiography, The Pleasures of Statistics, Frederick Mosteller gives an amusing example of why observational studies are no substitute for doing experiments.
We are all familiar with the idea that we can estimate height in male adults from their weight. … But not one of us believes that adding 20 pounds by eating and minimizing exercise will add an inch to our height.
The problem is not simply that the direction of causality backward, it’s that we cannot use a static description to predict what will happen if we change something.
Although regression situations may give one the illusion of finding out what would happen if we changed something, in the absence of an experiment they offer merely offer guesses.
He summarizes his point by quoting George Box:
To find out what happens to a system when you interfere with it, you have to interfere with it (and not just passively observe it).
Remember this next time you hear claims such as every dollar spent on X saves so many dollars spent on Y. Or every minute spent exercising increases your life expectancy by so many minutes. Or every time you do some activity you increase or decrease your risk of cancer by so much. First of all, these kinds of statements are linear extrapolations on situations that are not linear. Second, they may be observations that do not describe what will happen when you change something. They may be no more true than the idea that gaining weight makes you taller.
Here’s an example of how observation and intervention differ. Lottery winners often go bankrupt within a couple years of receiving their prize. If you suddenly make someone a millionaire, they’re not a typical millionaire.
Related posts:
Numerator-only data
Randomized trials of parachute use
{ 1 trackback }
{ 7 comments… read them below or add one }
John S. 03.12.10 at 09:01
Bravo. That is a great way of explaining “correlation is not causation”. I have already put Mosteller’s book on my wish list.
Vishal 03.12.10 at 10:23
Great post! Thanks.
Re. John S’s comment above, I don’t think this post is about “correlation is not causation”. Even if there’s causation in the data, I think one would still have to introduce and test a ‘change’ in the system to see what happens when you introduce that chance in the system. No?
John 03.12.10 at 10:37
Vishal, I believe there is more going on than correlation and causation. In the lottery example, influence goes both ways between wealth on the one hand and attitudes and circumstances on the other. But suddenly making someone a millionaire does not give them the typical characteristics of a millionaire. Also, a bankrupt former millionaire isn’t going to be a typical poor person.
ivanskie 08.05.10 at 14:17
[Editor's note: I thought the following spam comment was funny, so I included it minus the link they included. -- John]
great post! i have been looking for helpful ideas on how to gain weight in a healthy manner and i think your posts are relevant. i also found another useful website <spam URL> that has a lot of great information.
Rick Wicklin 05.11.11 at 12:31
Your comment about the “typical millionaire” and the “typical poor person” is important, too, and often misrepresented in the media. If every X hours of exercise really added Y minutes to my lifespan, I wouldn’t be sitting at this computer. And if every cigar smoked subtracted Q minutes from your lifespan, George Burns wouldn’t have lived to 100.
Steven H. Noble 11.21.11 at 13:54
On a related note I’d love to hear your thoughts on debiasing methods (like Heckman, etc). I say related because sometimes even interfering with the system isn’t sufficient to know the effects of interfering with the system. Like when you interfere in a biased way. The behaviour of lottery winners perhaps does not say all that much on the general effects of suddenly making someone a millionaire. Only on the effects of making a lottery player a millionaire.
rehabprof 11.22.11 at 04:59
I have heard it said that we are 17 times more likely to be struck with lightening than to win the lottery. But when we go outside in the rain ee say to ourselves, “it will never happen to me.”. When we buy a lottery ticket, we say, “Somebody has to win!”. Attitude trumps statistics.