Accuracy, precision, and recall

I posted the definitions of accuracy, precision, and recall on @BasicStatistics this afternoon. I think the tweet was popular because people find these terms hard to remember and they liked a succinct cheat sheet.

There seems to be no end of related definitions, and multiple names for the same definitions.

Precision is also known as positive predictive value (PPV) and recall is also known as sensitivityhit rate, and true positive rate (TPR).

Not mentioned in the tweet above are specificity (a.k.a. selectivity and true negative rate or TNR), negative predictive value (NPV), miss rate (a.k.a. false negative rate or FNR), fall-out (a.k.a. false positive rate or FPR), false discovery rate (FDR), and false omission rate (FOR).

How many terms are possible? There are four basic ingredients: TP, FP, TN, and FN. So if each term may or may not be included in a sum in the numerator and denominator, that’s 16 possible numerators and 16 denominators, for a total of 256 possible terms to remember. Some of these are redundant, such as one (a.k.a. ONE), given by TP/TP, FP/FP, etc. If we insist that the numerator and denominator be different, that eliminates 16 possibilities, and we’re down to a more manageable 240 definitions. And if we rule out terms that are the reciprocals of other terms, we’re down to only 120 definitions to memorize.

But wait! We’ve assumed every term occurs with a coefficient of either 0 or 1. But the F1 score has a couple coefficients of 2:

F1 = 2TP / (2TP + FP + FN)

If we allow coefficients of 0, 1, or 2, and rule out redundancies and reciprocals …

This has been something of a joke. Accuracy, precision, and recall are useful terms, though I think positive predictive value and true positive rate are easier to remember than precision and recall respectively. But the proliferation of terms is ridiculous. I honestly wonder how many terms for similar ideas are in use somewhere. I imagine there are scores of possibilities, each with some subdiscipline that thinks it is very important, and few people use more than three or four of the possible terms.

Here are the terms I’ve collected so far. If you know of other terms or other names for the same terms, please leave a comment and we’ll see how long this table goes.

Accuracy (TP + TN) / (FP + TP + FN + TN)
True positive rate (TPR) TP / (TP + FN)
Sensitivity see TPR
Recall see TPR
Hit rate see TPR
Probability of detection see TPR
True negative rate (TNR) TN / (TN + FP)
Specificity see TNR
Selectivity see TNR
Positive predictive value (PPV) TP / (TP + FP)
Precision see PPV
Negative predictive value (NPV) TN / (TN + FN)
False negative rate (FNR) FN / (FN + TP)
Miss rate see FNR
False positive rate (FPR) FP / (FP + TN)
Fall out see FPR
False discovery rate (FDR) FP / (FP + TP)
False omission rate (FOR) FN / (FN + TN)
F1 score 2TP / (2TP + FP + FN)
F-score see F1 score
F-measure see F1 score
Sørensen–Dice coefficient see F1 score
Dice similarity coefficient (DSC) see F1 score

4 thoughts on “Accuracy, precision, and recall

  1. Worse than multiple names for the same concept is different concepts described by the same name. One such is “sensitivity”, which can mean TPR; it can also refer to d-prime which is calculated by the following z-scores z(TPR) – z(FPR).

    Also worth noting that TPR = 1 – FNR and FPR = 1 – TNR. So, really, only two such rates are needed to completely characterize a response.

  2. As a purely combinatorial exercise, this is quite interesting. We might start with a simpler version of the problem with only two variables, A and B, and coefficients of 0 or 1. In principle there are 16 statistics of this type. However, 10 of them are identically either 0, 1, infinity, or NaN. The other 6 are:
    A/B, (A+B)/B, B/(A+B), A/(A+B), (A+B)/A, B/A
    I’ve deliberately ordered these so that each pair of consecutive statistics can easily be seen to be equivalent: either reciprocals or a sum or difference of 1. Hence, there is essentially only one statistic, or two if you count all the constants as one.

    This suggests that there are far fewer than 4^n essentially distinct statistics for n variables. A lower bound is 2^(n-1), since it is easy to see that A_1/(A_1 + … + A_n), A_2/(A_1 + … + A_n), …, A_(n-1)/(A_1 + … + A_n) and sums of any of these terms are essentially distinct.

  3. It is strange why TNR is not miss rate, where TPR is a hit rate. I think, whoever first named miss rate and hit rate, want to have miss rate and hit rate sum up to 1. But it doesn’t have to be. Miss rate should be TNR, to indicate the probability of missing correctly, i.e, if True was hit then False should be miss.

Leave a Reply

Your email address will not be published. Required fields are marked *