Statistical evidence versus legal evidence

This is the job of a juror in the US legal system described in statistical terms:

Compute the posterior probability of a defendant’s guilt conditioned on the admissible evidence, starting with a prior biased toward innocence. Report “guilty” if the posterior mean probability of guilt is above a level referred to as “beyond reasonable doubt.”

A juror is not to compute a probability conditioned on all evidence, only admissible evidence. One of the purposes of voir dire is to identify potential jurors who do not understand this concept and strike them from the jury pool. Very few jurors would understand or use the language of conditional probability, but a competent juror must understand that some facts are not to be taken into consideration in reaching a verdict.

For example, the fact that someone has been arrested, indicted by a grand jury, and brought to trial is not itself to be considered evidence of guilt. It is not legal evidence, but it certainly is statistical evidence: People on trial are more likely to be guilty of a crime than people who are not on trial.

This sort of schizophrenia is entirely proper. Statistical tendencies apply to populations, but trials are about individuals. The goal of a trial is to make a correct decision in an individual case, not to make correct decisions on average.[1]  Also, the American legal system embodies the belief that false positives are much worse than false negatives. [2]

Thinking of a verdict as a conditional probability allows a juror to simultaneously believe personally that someone is probably guilty while remaining undecided for legal purposes.

Related: A statistical problem with “nothing to hide”


[1] Jury instructions are implicitly Bayesian rather than frequentist in the sense that jurors are asked to come up with a degree of belief. They are not asked to imagine an infinite sequence of similar trials etc.

[2] For example, Benjamin Franklin said  “That it is better 100 guilty Persons should escape than that one innocent Person should suffer, is a Maxim that has been long and generally approved.” In decision theory vocabulary, this is a highly asymmetric loss function.

24 thoughts on “Statistical evidence versus legal evidence

  1. Speaking of inadmissible evidence: I’d be really curious how often it actually does sway juries. Say the murder weapon shows up with the victims blood and the suspects finger prints but the search was illegal. Is the jury really going to just ignore that evidence because the judge told them too? Should they?

    I don’t think there should be any such thing as inadmissible evidence. There can be planted evidence, counterfeited evidence etc but otherwise the evidence is the evidence. If procedures or the law was violated in acquiring the evidence bring up the police on charges. Instead what we have is the police try to get away with it and if they don’t 9/10 the evidence gets thrown out and the cop keeps his job. How is that helping protect peoples right to fair trial/expectation of privacy.

    Let the police illegally search you if they want. If you are guilty you go away, but the cop does too. If they want to give up say 3 years of their life to prove that you did it let them.

  2. The rules of evidence generally favor the defendant, and so they are in keeping with the value that it is better to let the guilty go than to incarcerate the innocent. In addition to protecting the defendant, they protect the society at large by providing disincentives for collecting certain kinds of evidence.

    The way I see it, a juror has two responsibilities: to render a verdict, and to uphold the rule of law. When a jury reluctantly lets a criminal go free on a technicality, they are defending the rights of future citizens by upholding the system.

  3. ‘People on trial are more likely to be guilty of a crime than people who are not on trial.’

    How do you know?

    Also, how do you decide the threshold of reasonable doubt?

  4. Of course, a very competent juror would know that a human brain cannot compute with only admissible evidence: we are just not built for that.

    We will take all the evidence we know into account (even inadmissible evidence).

  5. badmax: Most criminal defendants are found guilty, despite the fact that the legal system is deliberately biased in favor of their innocence. And juries require a unanimous decision, not a simple majority.

    There are mistakes, but it’s safe to say that most people found guilty are guilty. So if you condition on the fact that someone is on trial, and no other information, it’s highly likely that they’re guilty.

  6. @John the thing is most (at least of Law and Order and the like are to be believed) reasons for throwing out evidence isn’t that it is real evidence it was how it was obtained (illegal search and seizure etc). So it is illegal because law enforcement did it. However if your house was robbed and the police caught the guy and looked through things and saw the murder weapon (or the vandal just through it out the window onto the lawn) it would be far game.

    If it is illegal search treat it as such and actually prosecute the police. But in my mind it still doesn’t justify making the evidence “go away”. The process would still be protected illegal search would still be illegal and cops that value their freedom/job wouldn’t do it.

  7. Not all trials use the “beyond a reasonable doubt” standard of evidence for criminal cases. A “preponderance of the evidence” (posterior probability greater than .5) is also common. I don’t know how frequently jurors, vs. judges, get to decide such cases, but divided fractional responsibility is also common — a driver only 75% responsible for a collision may only be responsible for 75% of damages — where there isn’t just a boolean judgement, but an actual quantitative assignment of blame must be made.

  8. Criminal trials do not use Bayesian rules; neither do they use frequentist rules. The opposite of “frequentist” isn’t really “Bayesian”; rather, it’s “subjectivist” (and that’s not a slur). The problem is a court case is not to determine how likely a crime is; the problem is how likely THIS PERSON is to have committed THIS crime. Thus, subjectivist probabilities are required; frequentist probabilities are not useful, since the sample size is too small.

    Bayes’ equations provide a clear mathematical basis for correcting subjective opinions based on evidence, so subjectivist probabilities have to measure themselves against Bayesian math; but courts use a much, much older system and are not actually measured that way. I think it would be wonderful if courts would use Bayesian rules rather than the older ruleset they use.

    The general theory that courts use is now called “abductive reasoning”, or “inference to the best explanation”. That’s a VERY general term, and there’s a lot of liberty — you can basically get away with anything the judge, defense, and prosecutor all allow you to get away with.

  9. Quantum Mechanic

    @John: I agree with your main point of legal vs. statistical evidence.


    ‘People on trial are more likely to be guilty of a crime than people who are not on trial.’

    You don’t have statistics on guilt of people not on trial. (If you do, congratulations, you’ve just won your pick of tenure at a number of math and legal colleges. ;)

    I think what you mean to say is something like ‘People on trial are more likely to be guilty than not’, probably because the legal system is something of a market, and prefers to try those cases they think they can win. It costs a lot of resources to get to a trial, and a lot more to hold one.

    But you cannot compare guilt of those on trial to those not, unless you have some means of assessing guilt for the non-tried population. And you really shouldn’t use “verdict=guilt”, because of errors in verdict, unless there’s an estimate of verdict errors.

  10. SteveBrooklineMA

    The Constitution guarantees us a trial by an impartial jury. It doesn’t say anything (does it?) about Judges and/or Congress having the authority to tell the jury what it can and cannot take into account. I wonder if over time we have allowed the executive and legislative branch to assume a power to which they are not entitled.

  11. You sometimes hear the maxim “Better for a hundred guilty men to be set free than for one innocent man to be convicted.” This suggests that the jury’s task is frequentist not Bayesian — it is judged not with respect to any individual trial, but collectively over a series of trials of innocent (respectively guilty) defendants.

  12. @David: that maxim also assumes that the damage done to society by one person having a miserable life + loss in confidence in the system (assuming that the persons innocence ever gets discovered/admitted) is greater than letting a 100 nutjobs go free (and potentially continue to do their crimes). The problem with following maxims: once people throw them out there people stop thinking.

    Is it really worse to ban one guy from being near schools wrongly than to let 100 pedophiles go free? What is the damage done one way versus the other?

  13. Mike, this argument has never been just about weighing false positives versus false negatives. A second, perhaps even more important reason for strict rules about admissibility of evidence is that there is a long history of abuse—for political reasons, for personal reasons, or simply because of ingrained racism. Evidence is thrown out when it is obtained illegally to punish a historically common abuse of power. Finally, when the false positives and negatives are weighed, it is typically on ethical as well as practical grounds and rarely is the ratio as low as 1:100 as you are suggesting.

  14. @Krzysztof the problem is it really isn’t that much of a punishment. Yeah, they are frustrated they can’t get away with the evidence they got in a bad way. But do they lose their jobs (not optionally but actual policy enforcing this), do they lose their freedom? Letting someone break the law and then just taking away the ill gotten gains doesn’t discourage the behavior enough. It would be like if the only thing a police officer could do to a car thief is take the car back to the rightful owner.

    The 1/100 was from an earlier quote mentioned in the thread. I agree there are probably a much larger number of false positives and probably a wide variance from country to country (or race to race).

  15. Oh to be clear I mean how race vs race are treated by a system not trying to imply that justice enforcement varies by race/region of the world based on the race of the majority/those in power.

  16. @Mike,

    I would be happy to see an oversight agency with police powers charged with prosecuting abuses of administrative/police powers and laws to give such an agency teeth. At first I thought the FBI could simply take that role, but they too need some oversight.

  17. A judge in my remote acquaintance reports that he is usually pointed in instructing jurors NOT to construe “beyond reasonable doubt” as requiring some numerical probability; some legal historians in fact believe that the phrase is meant to ease a verdict of guilt, oweing to the sometime pattern that juries were hung (or acquitted imprudently) over genuine and persistent doubts that, nonetheless, were unreasonable.

  18. More generally, credibility (how much a claim ought to be believed) and relative frequency (the numerical proportion of a class that have some property) are entirely different concepts. Yet the word ‘probability’ is used for both, and it often misleads us.

    Personally, I think the “subjective” nature of credibility makes it entirely non-numerical, and so rules out the use of formalisms such as Bayes’ equations.

    I think it’s quite right to focus on the fact that a jury decides on individual cases rather than drawing on or arriving at statistical claims about pluralities of any sort.

  19. If you reject quantifying credibility, then credibility and probability are different. If you do quantify credibility, then subject to certain reasonable axioms, Cox’s theorem says that you get a system that is isomorphic to probability.

    It’s fine to say that credibility isn’t quantifiable. And it’s fine to say that credibility is quantifiable and obeys the laws of probability. But an intermediate position, saying that it is quantifiable but not the same as probability, has to contradict one of Cox’s axioms and strain common sense.

  20. I would stop using the word ‘probability’ for credibility, as I think it’s unquantifiable, but unfortunately most historical figures writing about epistemology (such as Hume) used the word ‘probability’. And unfortunately many present-day writers (including physicists) aren’t clear on the distinction.

  21. Contrary to this: “Personally, I think the ‘subjective’ nature of credibility makes it entirely non-numerical, and so rules out the use of formalisms such as Bayes’ equations.”

    Bayes’ equations are a formalization of what is called “subjective probability.” It is subjective because it depends not exclusively on the objective events being studied, but on the priors chosen by the investigator in order to begin the investigation. A less formal version of subjective probability is to ask an expert how likely they think the matter is.

    Subjectivity isn’t bad — in the case of Bayesian statistics, it’s merely an admission that we haven’t experienced everything, and we could be surprised.

    Now, I’m not sure what “credibility” might mean. Is it a formal term in common use? If not, the meaning I think of is the strength of my expectation that a given statement will correspond to reality. That could be computed by frequentist means (by counting the statements with known reality correspondance in the population) or by subjectivist means (whether by Bayesian means or by asking an expert).

  22. I think the idea of “priors” is is non-applicable to the individual, non-statistcial hypotheses that juries make decisions about. (And indeed which guide everyday decisions in life, and theory choice in science.)

    That’s because the “credibility” of a claim — “how much it ought to be believed”, its “epistemic justification”, or whatever we choose to call it — depends on the other beliefs that are already in the mind of the person contemplating it. These differ from one individual to the next, and even from one moment to the next.

    Assigning “prior probabilities” to the hypotheses that come into play when juries make decisions wouldn’t just be conceptually confused — it would further mislead by disguising vital differences between individuals’ belief systems. In effect, it imposes a rigid orthodoxy on what should or should not be considered seriously.

    There is a sort of “traditional wisdom” in having juries consist of a dozen or so people from different walks of life. Ideally, that’s large enough to ensure a heterodoxy among those making the decision, yet small enough to enable them to discuss the issues among themselves. With the sort of issues that juries have to decide on, discussion rather than numbers rule — and should rule.

  23. Bowman, I definitely agree that juries should discuss the issue, and also that the range of experiences they offer is part of their benefit. I don’t agree that any of this makes Bayesian statistics unuseful in court. I don’t think Bayesian statistics is always useful, though.

    The existence of prior beliefs doesn’t invalidate Bayesian statistics; in fact, that’s the reason for Bayesian statistics. Imagine (purely for the sake of argument, not a real proposal) a courtroom in which there was a central scoreboard showing Bayesian statistics based on arguments currently being presented and the priors required by the judge’s instructions. Now imagine that one of the jurors sees that the numbers, after a new argument is presented, fails to match his idea of the probabilities. That juror then has a chance to think about WHY the numbers don’t match intuition, and should therefore be instructed that when something looks wrong, he should introspect to see what in his background gave him an insight into the case — therefore allowing him to see what the normal procedures of the court couldn’t. (End of thought-experiment, and let me add that this is more like deliberation and less like a trial. Actually, I suppose that a jury foreman could choose to actually DO this.)

    “it imposes a rigid orthodoxy on what should or should not be considered seriously.” That’s what happens in courts. That’s what’s SUPPOSED to happen. I think that some exclusionary rules are bad (I don’t think that cops should be punished by freeing a criminal); but the foundation is solid, that some facts are prejudicial for example because they confuse correlation with causation, or because they’re obviously cherry-picked.

Comments are closed.