Comments on: Four types of errors http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/ The blog of John D. Cook Sat, 11 Feb 2012 01:10:06 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Jerzy http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-120826 Jerzy Fri, 09 Dec 2011 17:00:22 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-120826 I really like the idea of Type S error but am having trouble thinking through how to apply it in practice. Say you expect θj and θk to be pretty close, but certainly not identical. So you'd like to be able to say one of these three things: "θj is bigger than θk," "θk is bigger than θj," or "We don't have enough evidence to decide which is bigger." Then, let's say you do what many people do in practice. Perform a standard test of H0: θj=θk at confidence level 0.05. Then either you fail to reject H0, saying "I don't have enough evidence to decide which one is the bigger one"... or you do reject H0, then say "They are statistically significantly different and θj has the higher point estimate so I am confident that θj is bigger than θk" (or vice versa, depending). Can we say anything precise about our probability of Type S error under this procedure? I really like the idea of Type S error but am having trouble thinking through how to apply it in practice.
Say you expect θj and θk to be pretty close, but certainly not identical. So you’d like to be able to say one of these three things: “θj is bigger than θk,” “θk is bigger than θj,” or “We don’t have enough evidence to decide which is bigger.”
Then, let’s say you do what many people do in practice. Perform a standard test of H0: θj=θk at confidence level 0.05. Then either you fail to reject H0, saying “I don’t have enough evidence to decide which one is the bigger one”… or you do reject H0, then say “They are statistically significantly different and θj has the higher point estimate so I am confident that θj is bigger than θk” (or vice versa, depending).
Can we say anything precise about our probability of Type S error under this procedure?

]]> By: Type R error — The Endeavour http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-120769 Type R error — The Endeavour Fri, 09 Dec 2011 13:01:28 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-120769 [...] Gelman added a couple more types of error to the standard repertoire of type I and type II errors. He suggests using type S [...] [...] Gelman added a couple more types of error to the standard repertoire of type I and type II errors. He suggests using type S [...]

]]> By: jean-louis http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-110188 jean-louis Tue, 25 Oct 2011 19:36:17 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-110188 Hi John, <cite>“significantly different” is related to the strength of evidence of a difference, not its size of the difference</cite> I agree, but I did not say that, did I? I guess you can measure the size of the difference with a hypothesis like θj = θk+epsilon. With an inequality hypothesis like θj > θk you do not solve this problem, either. The type M error would do, but I would need to investigate to know more what it is all about :-) Hi John,
“significantly different” is related to the strength of evidence of a difference, not its size of the difference
I agree, but I did not say that, did I? I guess you can measure the size of the difference with a hypothesis like θj = θk+epsilon. With an inequality hypothesis like θj > θk you do not solve this problem, either.

The type M error would do, but I would need to investigate to know more what it is all about :-)

]]> By: John http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-110184 John Tue, 25 Oct 2011 19:17:56 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-110184 jean-louis: In statistics, "significantly different" is related to the strength of evidence of a difference, not its size of the difference. The null hypothesis is typically that two treatments have exactly the same effect. If there is statistically significant data that the null is false, that doesn't mean that there is a large difference in the effects, only a large amount of evidence that there is a non-zero difference. You could have strong evidence of a small effect. jean-louis: In statistics, “significantly different” is related to the strength of evidence of a difference, not its size of the difference. The null hypothesis is typically that two treatments have exactly the same effect. If there is statistically significant data that the null is false, that doesn’t mean that there is a large difference in the effects, only a large amount of evidence that there is a non-zero difference. You could have strong evidence of a small effect.

]]> By: jean-louis http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-110182 jean-louis Tue, 25 Oct 2011 19:01:49 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-110182 Hi John, <cite>The point is that no two treatments are ever identical.</cite> While I would agree to this, I believe that it still is not in contradiction with the hypothesis θj = θk, in which the θs do not replace the "treatments" (which, for the experiments to be useful, have to be different!), but rather, usually, the effects of the treatments. I am not an expert on the topic, nor an expert on statistics, so you might need to correct me if I m wrong. How I understand the null hypothesis is: "are the effects of treatment A statistically different from those of treatment B?" and usually, you don't really get a clear answer, but rather a measure telling you how likely it is that they are different (and hopefully medical publications are putting great care in keeping this in mind). As for type S or M errors, I am not sure how that would work. However, it made me think of the difference between one-tailed and two-tailed hypothesis test. As I understand, type S or M would still be type I or II errors, but given different kind of hypothesis (inequality or equality, demonstration is left as homework ;-)). Or am I completely off-topic? even more than all the above spam about spam ?!! (no offense, just thought about the funny connection...) Hi John,
The point is that no two treatments are ever identical.
While I would agree to this, I believe that it still is not in contradiction with the hypothesis θj = θk, in which the θs do not replace the “treatments” (which, for the experiments to be useful, have to be different!), but rather, usually, the effects of the treatments.

I am not an expert on the topic, nor an expert on statistics, so you might need to correct me if I m wrong. How I understand the null hypothesis is: “are the effects of treatment A statistically different from those of treatment B?” and usually, you don’t really get a clear answer, but rather a measure telling you how likely it is that they are different (and hopefully medical publications are putting great care in keeping this in mind).

As for type S or M errors, I am not sure how that would work. However, it made me think of the difference between one-tailed and two-tailed hypothesis test. As I understand, type S or M would still be type I or II errors, but given different kind of hypothesis (inequality or equality, demonstration is left as homework ;-) ).

Or am I completely off-topic? even more than all the above spam about spam ?!! (no offense, just thought about the funny connection…)

]]> By: Andy http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76689 Andy Mon, 18 Apr 2011 08:38:01 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76689 Roman: thanks! Roman: thanks!

]]> By: Roman Cheplyaka http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76674 Roman Cheplyaka Mon, 18 Apr 2011 03:54:37 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76674 Andy: for example, here are the features that SpamAssassin tests for: http://wiki.apache.org/spamassassin/RulesList (They are separate from the Bayes classifier and are more about meta-information than the contents.) Andy: for example, here are the features that SpamAssassin tests for: http://wiki.apache.org/spamassassin/RulesList
(They are separate from the Bayes classifier and are more about meta-information than the contents.)

]]> By: Andy http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76604 Andy Sun, 17 Apr 2011 21:40:15 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76604 What spam filters do could be viewed in terms of a, say, logistic regression model predicting probability an email is spam as a function of a bunch of features, e.g., presence of a particular word. Then, you've got a slope, estimated using seen emails (sample) for each feature. Null hypothesis: slope = 0, for each slope. Type S error would be inferring that a feature is indicative of spam when it's indicative of a safe email or vice versa. Type M error would be, e.g., inferring that the presence of a particular feature is more likely to indicate spam than it really is in the broader population of emails. Back to the oldies: a type 1 error would be inferring, based on sample emails, that a feature is predictive of spam (i.e., the slope for the feature is statistically significantly different to zero) when it's not in the population of emails. Type 2: you think based on sample that a feature is not predictive when in the population it is. I guess the notion of population is quite tricky for emails, though! Spam detection is a really nice example. Actually, has anyone looked at what features tend to predict whether an email is spam, or is it all kept secret and/or hidden in the CPTs of a Bayesian classifier somewhere? What spam filters do could be viewed in terms of a, say, logistic regression model predicting probability an email is spam as a function of a bunch of features, e.g., presence of a particular word. Then, you’ve got a slope, estimated using seen emails (sample) for each feature. Null hypothesis: slope = 0, for each slope.

Type S error would be inferring that a feature is indicative of spam when it’s indicative of a safe email or vice versa. Type M error would be, e.g., inferring that the presence of a particular feature is more likely to indicate spam than it really is in the broader population of emails.

Back to the oldies: a type 1 error would be inferring, based on sample emails, that a feature is predictive of spam (i.e., the slope for the feature is statistically significantly different to zero) when it’s not in the population of emails. Type 2: you think based on sample that a feature is not predictive when in the population it is.

I guess the notion of population is quite tricky for emails, though! Spam detection is a really nice example.

Actually, has anyone looked at what features tend to predict whether an email is spam, or is it all kept secret and/or hidden in the CPTs of a Bayesian classifier somewhere?

]]> By: John http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76296 John Fri, 15 Apr 2011 21:28:10 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76296 Andrew does seem to assume all null hypotheses are point hypotheses. I suppose he does this because so often that is the case in practice, even though in theory a null hypothesis could be any arbitrary subset of the parameter space. Andrew does seem to assume all null hypotheses are point hypotheses. I suppose he does this because so often that is the case in practice, even though in theory a null hypothesis could be any arbitrary subset of the parameter space.

]]> By: Roman Cheplyaka http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76293 Roman Cheplyaka Fri, 15 Apr 2011 21:20:51 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76293 <q>A spam filter does not have a point null hypothesis.</q> Exactly. I was under a (probably wrong) impression that you or Andrew Gelman argue for somehow replacing all type 1/type 2 errors with type s/type m errors. A spam filter does not have a point null hypothesis.
Exactly. I was under a (probably wrong) impression that you or Andrew Gelman argue for somehow replacing all type 1/type 2 errors with type s/type m errors.

]]> By: John http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76290 John Fri, 15 Apr 2011 21:13:49 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76290 A spam filter does not have a point null hypothesis. Type-S error is relevant if you think of a spaminess scale with 0 being neutral and increasing values corresponding to more offensive spam. A spam filter does not have a point null hypothesis. Type-S error is relevant if you think of a spaminess scale with 0 being neutral and increasing values corresponding to more offensive spam.

]]> By: Roman Cheplyaka http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76285 Roman Cheplyaka Fri, 15 Apr 2011 20:44:57 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76285 Often Type 1/Type 2 errors (or "false positive"/"false negative") make more sense. E.g. how would you formulate spam detection errors in terms of "Type M" and "Type S"? Often Type 1/Type 2 errors (or “false positive”/”false negative”) make more sense. E.g. how would you formulate spam detection errors in terms of “Type M” and “Type S”?

]]> By: John http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76268 John Fri, 15 Apr 2011 18:39:12 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76268 Thanks. I updated the link. Thanks. I updated the link.

]]> By: Mathew Woodyard http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/comment-page-1/#comment-76265 Mathew Woodyard Fri, 15 Apr 2011 18:29:13 +0000 http://www.johndcook.com/blog/2008/04/21/four-types-of-errors/#comment-76265 Great post, as always, but I noticed that your link to Gelman's presentation is broken. It looks like it has moved here: http://www.stat.columbia.edu/~gelman/presentations/multiple_minitalk2.pdf Great post, as always, but I noticed that your link to Gelman’s presentation is broken. It looks like it has moved here: http://www.stat.columbia.edu/~gelman/presentations/multiple_minitalk2.pdf

]]>