Continuum between anecdote and data

The difference between anecdotal evidence and data is overstated. People often have in mind this dividing line where observations on one side are worthless and observations on the other side are trustworthy. But there’s no such dividing line. Observations are data, but some observations are more valuable than others, and there’s a continuum of value.

Rib eye steak

I believe rib eye steaks are better for you than rat poison. My basis for that belief is anecdotal evidence. People who have eaten rib eye steaks have fared better than people who have eaten rat poison. I don’t have exact numbers on that, but I’m pretty sure it’s true. I have more confidence in that than in any clinical trial conclusion.

Hearsay evidence about food isn’t very valuable, per observation, but since millions of people have eaten steak for thousands of years, the cumulative weight of evidence is pretty good that steak is harmless if not good for you. The number of people who have eaten rat poison is much smaller, but given the large effect size, there’s ample reason to suspect that eating rat poison is a bad idea.

Now suppose you want to get more specific and determine whether rib eye steaks are good for you in particular. (I wouldn’t suggest trying rat poison.) Suppose you’ve noticed that you feel better after eating a steak. Is that an anecdote or data? What if you look back through your diary and noticed that every mention of eating steak lately has been followed by some remark about feeling better than usual. Is that data? What if you decide to flip a coin each day for the next month and eat steak if the coin comes up heads and tofu otherwise. Each of these steps is an improvement, but there’s no magical line you cross between anecdote and data.

Suppose you’re destructively testing the strength of concrete samples. There are better and worse ways to conduct such experiments, but each sample gives you valuable data. If you test 10 samples and they all withstand two tons of force per square inch, you have good reason to believe the concrete the samples were taken from can withstand such force. But if you test a drug on 10 patients, you can’t have the same confidence that the drug is effective. Human subjects are more complicated than concrete samples. Concrete samples aren’t subject to placebo effects. Also, cause and effect are more clear for concrete. If you apply a load and the sample breaks, you can assume the load caused the failure. If you treat a human for a disease and they recover, you can’t be as sure that the treatment caused the recovery. That doesn’t mean medical observations aren’t data.

Carefully collected observations in one area may be less statistically valuable than anecdotal observations in another. Observations are never ideal. There’s always some degree of bias, effects that can’t be controlled, etc. There’s no quantum leap between useless anecdotes and perfectly informative data. Some data are easy to draw inference from, but data that’s harder to understand doesn’t fail to be data.

10 thoughts on “Continuum between anecdote and data

  1. There’s also the matter of method & community norms. Observations made in accordance with what is considered acceptable practice by the community that the observer belongs to will tend to carry more weight than observations that do not. In this sense, the former will be “valid observational data” and the latter “anecdotal” – which may be arbitrary but such protocols are presumably based on trial and error correction of what can result in unreliable data. The recent issues with replication of psychological studies suggest the fragility (and importance) of such norms.

  2. Not that this takes away from your point, but you chose a funny example. I’m pretty sure there are lots of people whose cardiologists tell them that they should consume rat poison and shouldn’t consume rib eye steaks.

  3. I completely agree. Also, I apologize in advance for some darker reflections.

    I frequently read about experiments that involve non-human animals, which caused obvious harm only to reach obvious conclusions (or conclusions of doubtful value). I certainly believe in science and the empirical method, but that’s no excuse for ignoring the obvious.

  4. Also, despite my love for human and non-human animals, I have to say that I’ve moved on from the occasional rib eye… to the occasional t-bone.

  5. I agree that the difference in meaning may be overstated, but I’m not sure there’s a perfect continuum between anecdote and data. For me, the distinguishing features of anecdotes vs data are the anecdotes’ absence of any control or comparison and the lack of any quantitative information on incidence. Things I regard as data usually come with at least one of those – both if experimental – and I think that sharply differentiates anecdote.
    Of course we can have _poor_ data that is not a heck of a lot better than a pile of anecdotes (badly chosen controls, selective counts, biased selection … you name it, it’s happened) and there I think there _is_ a continuum from the poor to almost the unreachable perfect. I say ‘unreachable’ because we will probably never see any data set without _some_ choice that influences the outcome – even if only the choice to collect it at all.

  6. My eyes immediately went to the photo, getting the title as a passing glance, which my mind saw as “The continuum between artichoke and data.”

    To me, “data” implies a degree of care in capture, either with direct control of the environment, or a careful characterization of it. Anecdote, more often used in the phrase “anecdotal evidence” is still evidence, merely with less control over or awareness of the context.

    Unfortunately, some information is only available anecdotally, which generally means far more of it is needed before any statistical meaning can be derived. And even then the analysis has to be exceedingly careful.

    The best and most common example is the rare glitch that occurs in a manufactured product, but which occurs only in the field, never in the lab. We first assume “operator error”, and flag it as “not reproducible”. Then another similar, but different, report arrives, and we wonder if it could possibly be related to that earlier report.

    But we still lack enough information to generalize to the point of formulating a useful theory that may explain the phenomenon. When a third vaguely similar report comes in, we then typically start a new lifetime test, where we take as many units as we can spare and start running them 24/7 in variable environments to cover the contexts of the reports.

    I’d say that identifies the problem less than 10% of the time, though other unrelated issues are likely to be discovered. At which point we decide that we have gone far beyond our minimum due diligence, and get our lab back to normal.

    We used to call such anecdotal failures ‘unicorns’, but we decided a better term was needed, one that sounded more professional should it be heard outside our premises.

    So we now call them “cosmic ray hits” instead, and cloak them in talk of Single Event Effects.

  7. John:

    I think you’re making a statistical error when you attribute your confidence in steak over rat poison to data. This seems to be the same error attributed to Laplace in the famous bet about whether the sun will rise tomorrow. As with Laplace’s beliefs about the sun, I think your beliefs about steak and rat poison are informed by theory. Rat poison is supposed to kill rats, people aren’t so different from rats, hence you don’t want to eat rat poison. You write, “given the large effect size, there’s ample reason to suspect that eating rat poison is a bad idea.” That’s theory for you! I don’t have any data at all on people who have eaten rat poison but I’m still not going to do it.

    I don’t disagree with your point on anecdote and data, I just am hyper-alert to people attributing beliefs to data which are actually coming from theory.

  8. Theory is certainly part of it. Beliefs are always informed by some combination of data and theory. Maybe theory alone, but never data alone.

  9. In some cases, rat poison may be better than steak.

    Warfarin is a drug used for preventing blood clots and is derived from a commonly used rat poison. It is used often in patients with heart disease for whom a high fat meal can be dangerous.

  10. Andrew, John: if the term “anecdote” in the title would be changed to “prior belief” or “prior knowledge”, we would be directly in the Bayesian world. And this is where I think we are when we discuss steaks vs. rat poison.

    Andrew comments on theory being intermingled with and at times difficult to discern from data. This leads directly to formulating prior knowledge or assumptions, respectively. The prior knowledge is something that is often a muddle of data not available but in one’s (subjective) memory.

    In turn, statisticians would want to assess the credibility of some sort of hypothesis, in this case that steak is better for you (for anyone??) than rat poison. Fortunately, no such experiment has yet been conducted. The scattered reports of people taking rat poison can be regarded as what they call nowadays “real-world evidence.”
    Interestingly enough, if I were to challenge you to produce scientifically sound evidence that eating steak is better than eating rat poison, you would presumably be hard pressed.

    So we are stuck with beliefs (priors) with very limited data. A speaker on real-world evidence comes to mind who stated that the parachute as a life-saving device has also never been tested in a randomized trial. Still, there is reason to believe that it is useful.

    Conclusion: living by using your beliefs can be a life-saver.

    Footnote: the salad on the photo obviously muddles up the experiment.

Comments are closed.