In his autobiography, The Pleasures of Statistics, Frederick Mosteller gives an amusing example of why observational studies are no substitute for doing experiments.
We are all familiar with the idea that we can estimate height in male adults from their weight. … But not one of us believes that adding 20 pounds by eating and minimizing exercise will add an inch to our height.
The problem is not simply that the direction of causality backward, it’s that we cannot use a static description to predict what will happen if we change something.
Although regression situations may give one the illusion of finding out what would happen if we changed something, in the absence of an experiment they offer merely offer guesses.
He summarizes his point by quoting George Box:
To find out what happens to a system when you interfere with it, you have to interfere with it (and not just passively observe it).
Remember this next time you hear claims such as every dollar spent on X saves so many dollars spent on Y. Or every minute spent exercising increases your life expectancy by so many minutes. Or every time you do some activity you increase or decrease your risk of cancer by so much. First of all, these kinds of statements are linear extrapolations on situations that are not linear. Second, they may be observations that do not describe what will happen when you change something. They may be no more true than the idea that gaining weight makes you taller.
Here’s an example of how observation and intervention differ. Lottery winners often go bankrupt within a couple years of receiving their prize. If you suddenly make someone a millionaire, they’re not a typical millionaire.
Bravo. That is a great way of explaining “correlation is not causation”. I have already put Mosteller’s book on my wish list.
Great post! Thanks.
Re. John S’s comment above, I don’t think this post is about “correlation is not causation”. Even if there’s causation in the data, I think one would still have to introduce and test a ‘change’ in the system to see what happens when you introduce that chance in the system. No?
Vishal, I believe there is more going on than correlation and causation. In the lottery example, influence goes both ways between wealth on the one hand and attitudes and circumstances on the other. But suddenly making someone a millionaire does not give them the typical characteristics of a millionaire. Also, a bankrupt former millionaire isn’t going to be a typical poor person.
[Editor’s note: I thought the following spam comment was funny, so I included it minus the link they included. — John]
great post! i have been looking for helpful ideas on how to gain weight in a healthy manner and i think your posts are relevant. i also found another useful website <spam URL> that has a lot of great information.
:-) I’m also looking for a little weight gain (too skinny!) and I wouldn’t mind some extra height either.
Your comment about the “typical millionaire” and the “typical poor person” is important, too, and often misrepresented in the media. If every X hours of exercise really added Y minutes to my lifespan, I wouldn’t be sitting at this computer. And if every cigar smoked subtracted Q minutes from your lifespan, George Burns wouldn’t have lived to 100.
On a related note I’d love to hear your thoughts on debiasing methods (like Heckman, etc). I say related because sometimes even interfering with the system isn’t sufficient to know the effects of interfering with the system. Like when you interfere in a biased way. The behaviour of lottery winners perhaps does not say all that much on the general effects of suddenly making someone a millionaire. Only on the effects of making a lottery player a millionaire.
I have heard it said that we are 17 times more likely to be struck with lightening than to win the lottery. But when we go outside in the rain ee say to ourselves, “it will never happen to me.”. When we buy a lottery ticket, we say, “Somebody has to win!”. Attitude trumps statistics.
Hi,
two comments. First, if I may paraphrase your larger point, is that it is crucial for the controlled study to reproduce the phenomenon of interest in all relevant aspects, and the wealth-by-lottery experiment does not. Unfortunately, that word “relevant” is ambiguous, and interpreted in the strictest way possible, it makes controlled experiments impossible, because, in reality, you don’t have control. One example could be that diets are much more effective when administered to inpatients, under strict control, then outpatients, where compliance is a major problem. But compliance is a much different proposition if a scientist tells you what to eat vs. you deciding what to eat. So we can never test the effectiveness of diets undertaken under the individual own control. By “controlling” we modify reality. What type of control results in informative experiments? I don’t think we really know. You may be interested in this article: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60563-1/fulltext
My other comment is that it’s easy to be dismissive of observational studies (for the very good reasons you mention) until one realizes that it implies denying complete branches of science: cosmology, climatology, a big fraction of the social sciences. For instance, how would you design a controlled experiment to corroborate the theory of the Big Bang, or that global warming is man-made?
Easy, just randomly sample half of the civilized planets using carbon based fuels and force them to stop. You see the practical problem with that. Which I think points to the complexity and unevenness in the methods of science (some academics coined the term “epistemic cultures”) that is often understated not only in the popular media but even by practitioners and that indirectly exposes science to the deniers and lunatics out there, and prevents progress on the methods and foundations. Thanks
A person is a member of the aristocracy, the middle class, or the poor. Class is a collection of behaviors including money habits. You’re the aristocracy if you live off past investments, rather than salary. You’re middle class if you save a little of our salary. You’re poor if you live on your last dime. Winning a lottery won’t change your class affiliation or behavior. Your class affiliation is very deep and hard to escape. It is not a matter of how big your pile of money happens to be.
Becoming a member of the aristocracy takes two generations. Your dad can put you there, but you can’t get there yourself no matter how much money you have.