I posted the definitions of accuracy, precision, and recall on @BasicStatistics this afternoon. I think the tweet was popular because people find these terms hard to remember and they liked a succinct cheat sheet.
Accuracy = (TP+TN)/(TP+FP+FN+TN)
Precision = TP/(TP+FP)
Recall = TP/(TP+FN)where
T = true
F = false
P = positive
N = negative— Basic Statistics (@BasicStatistics) September 6, 2018
There seems to be no end of related definitions, and multiple names for the same definitions.
Precision is also known as positive predictive value (PPV) and recall is also known as sensitivity, hit rate, and true positive rate (TPR).
Not mentioned in the tweet above are specificity (a.k.a. selectivity and true negative rate or TNR), negative predictive value (NPV), miss rate (a.k.a. false negative rate or FNR), fall-out (a.k.a. false positive rate or FPR), false discovery rate (FDR), and false omission rate (FOR).
How many terms are possible? There are four basic ingredients: TP, FP, TN, and FN. (You may see these arranged into a 2 by 2 grid called a confusion matrix.) So if each term may or may not be included in a sum in the numerator and denominator, that’s 16 possible numerators and 16 denominators, for a total of 256 possible terms to remember. Some of these are redundant, such as one (a.k.a. ONE), given by TP/TP, FP/FP, etc. If we insist that the numerator and denominator be different, that eliminates 16 possibilities, and we’re down to a more manageable 240 definitions. And if we rule out terms that are the reciprocals of other terms, we’re down to only 120 definitions to memorize.
But wait! We’ve assumed every term occurs with a coefficient of either 0 or 1. But the F1 score has a couple coefficients of 2:
F1 = 2TP / (2TP + FP + FN)
If we allow coefficients of 0, 1, or 2, and rule out redundancies and reciprocals …
This has been something of a joke. Accuracy, precision, and recall are useful terms, though I think positive predictive value and true positive rate are easier to remember than precision and recall respectively. But the proliferation of terms is ridiculous. I honestly wonder how many terms for similar ideas are in use somewhere. I imagine there are scores of possibilities, each with some subdiscipline that thinks it is very important, and few people use more than three or four of the possible terms.
Here are the terms I’ve collected so far. If you know of other terms or other names for the same terms, please leave a comment and we’ll see how long this table goes.
| Accuracy | (TP + TN) / (FP + TP + FN + TN) |
| True positive rate (TPR) | TP / (TP + FN) |
| Sensitivity | see TPR |
| Recall | see TPR |
| Hit rate | see TPR |
| Probability of detection | see TPR |
| True negative rate (TNR) | TN / (TN + FP) |
| Specificity | see TNR |
| Selectivity | see TNR |
| Positive predictive value (PPV) | TP / (TP + FP) |
| Precision | see PPV |
| Negative predictive value (NPV) | TN / (TN + FN) |
| False negative rate (FNR) | FN / (FN + TP) |
| Miss rate | see FNR |
| False positive rate (FPR) | FP / (FP + TN) |
| Fall out | see FPR |
| False discovery rate (FDR) | FP / (FP + TP) |
| False omission rate (FOR) | FN / (FN + TN) |
| F1 score | 2TP / (2TP + FP + FN) |
| F-score | see F1 score |
| F-measure | see F1 score |
| Sørensen–Dice coefficient | see F1 score |
| Dice similarity coefficient (DSC) | see F1 score |
Worse than multiple names for the same concept is different concepts described by the same name. One such is “sensitivity”, which can mean TPR; it can also refer to d-prime which is calculated by the following z-scores z(TPR) – z(FPR).
Also worth noting that TPR = 1 – FNR and FPR = 1 – TNR. So, really, only two such rates are needed to completely characterize a response.
The “Sensitivity and Specificity” Wikipedia page has a nice, logical (and colorful!) illustration of many of these concepts: https://en.wikipedia.org/wiki/Sensitivity_and_specificity
As a purely combinatorial exercise, this is quite interesting. We might start with a simpler version of the problem with only two variables, A and B, and coefficients of 0 or 1. In principle there are 16 statistics of this type. However, 10 of them are identically either 0, 1, infinity, or NaN. The other 6 are:
A/B, (A+B)/B, B/(A+B), A/(A+B), (A+B)/A, B/A
I’ve deliberately ordered these so that each pair of consecutive statistics can easily be seen to be equivalent: either reciprocals or a sum or difference of 1. Hence, there is essentially only one statistic, or two if you count all the constants as one.
This suggests that there are far fewer than 4^n essentially distinct statistics for n variables. A lower bound is 2^(n-1), since it is easy to see that A_1/(A_1 + … + A_n), A_2/(A_1 + … + A_n), …, A_(n-1)/(A_1 + … + A_n) and sums of any of these terms are essentially distinct.
It is strange why TNR is not miss rate, where TPR is a hit rate. I think, whoever first named miss rate and hit rate, want to have miss rate and hit rate sum up to 1. But it doesn’t have to be. Miss rate should be TNR, to indicate the probability of missing correctly, i.e, if True was hit then False should be miss.
Balanced accuracy = (PPV + NPV) / 2. Can be useful.