{"id":8310,"date":"2011-04-14T07:55:51","date_gmt":"2011-04-14T13:55:51","guid":{"rendered":"http:\/\/www.johndcook.com\/blog\/?p=8310"},"modified":"2022-10-20T22:55:14","modified_gmt":"2022-10-21T03:55:14","slug":"significance-testing-and-congress","status":"publish","type":"post","link":"https:\/\/www.johndcook.com\/blog\/2011\/04\/14\/significance-testing-and-congress\/","title":{"rendered":"Significance testing and Congress"},"content":{"rendered":"

The US Supreme Court’s criticism of significance testing has been in the news lately. Here’s a criticism of significance testing involving the US Congress. Consider the following syllogism.<\/p>\n

    \n
  1. If a person is an American, he is not a member of Congress.<\/li>\n
  2. This person is a member of Congress.<\/li>\n
  3. Therefore he is not American.<\/li>\n<\/ol>\n

    The initial premise is false, but the reasoning is correct if we assume the initial premise is true.<\/p>\n

    The premise that Americans are never members of Congress is clearly false. But it’s almost true! The probability of an American being a member of Congress is quite small, about 535\/309,000,000. So what happens if we try to salvage the syllogism above by inserting “probably” in the initial premise and conclusion?<\/p>\n

      \n
    1. If a person is an American, he is probably<\/strong> not a member of Congress.<\/li>\n
    2. This person is a member of Congress.<\/li>\n
    3. Therefore he is probably<\/strong> not American.<\/li>\n<\/ol>\n

      What went wrong? The probability is backward. We want to know the probability that someone is American given he is a member of Congress, not the probability he is a member of Congress given he is American.<\/p>\n

      Science continually uses flawed reasoning analogous to the example above. We start with a “null hypothesis,” a hypothesis we seek to disprove. If our data are highly unlikely assuming this hypothesis, we reject that hypothesis.<\/p>\n

        \n
      1. If the null hypothesis is correct, then these data are highly unlikely.<\/li>\n
      2. These data have occurred.<\/li>\n
      3. Therefore, the null hypothesis is highly unlikely.<\/li>\n<\/ol>\n

        Again the probability is backward. We want to know the probability of the hypothesis<\/em> given the data<\/em>, not the probability of the data<\/em> given the hypothesis<\/em>.<\/p>\n

        We can’t reject a null hypothesis just because we’ve seen data that are rare under this hypothesis. Maybe our data are even more rare under the alternative. It is rare for an American to be in Congress, but it is even more rare for someone who is not American to be in the US Congress!<\/p>\n

        I found this illustration in The Earth is Round (p < 0.05) by Jacob Cohen (1994). Cohen in turn credits Pollard and Richardson (1987) in his references.<\/p>\n

        More statistics posts<\/h2>\n