Precision and recall

What are precision and recall?

Precision and recall are two scores that measure how well a classification model finds a specific thing: fraud, spam, defects, churn, or whatever you have labelled as the positive class. They split the loose question is the model right? into two sharper ones that often have very different answers.

Precision asks one thing: of everything the model flagged, how much was actually right? Recall asks another: of everything that was really there, how much did the model catch? A model can score high on one and low on the other, so you almost never read one without the other.

Both are built from three counts. A true positive (TP) is a real case the model flagged. A false positive (FP) is a flag that turned out wrong. A false negative (FN) is a real case the model missed. Precision is TP divided by everything you flagged, the share of alerts that were correct. Recall is TP divided by everything that was really positive, the share of real cases you caught.

In machine learning these are the standard way to grade a classifier trained by supervised learning, especially when the positive class is rare or costly to get wrong. Fraud and medical screening are typical: one class matters far more than the other, so a single accuracy number would hide the mistakes you care about.

A worked example

Say a model reviews 1,000 card transactions. In reality 100 of them are fraud and 900 are legitimate. The model flags 120 as fraud. When you check those flags against what actually happened, the counts land like this: 90 were real fraud, 30 were legitimate transactions caught by mistake, 10 real fraud cases slipped through unflagged, and the other 870 legitimate transactions were correctly left alone. Those four counts together are a confusion matrix.

So TP is 90, FP is 30, and FN is 10. Drop them into the two formulas:

precision = TP / (TP + FP) = 90 / (90 + 30) = 90 / 120 = 0.75
recall    = TP / (TP + FN) = 90 / (90 + 10) = 90 / 100 = 0.90

Precision is 0.75, so 75 percent of the fraud alerts were real and one in four was a false alarm. Recall is 0.90, so the model caught 90 of the 100 real fraud cases and missed 10. Same model and data, two verdicts: it casts a wide net that catches most fraud but rings plenty of false alarms.

How the decision threshold trades one for the other

A classifier does not hand you a hard yes or no. It outputs a score, and you pick a decision threshold above which a case counts as positive. Move that threshold and both metrics move, in opposite directions.

Lower the threshold and the model flags more cases. It catches more real positives, so recall goes up, but it also lets more wrong flags through, so precision goes down. Raise the threshold and the reverse happens: the model only flags the clearest cases, precision climbs, and more real cases slip past, so recall drops. Precision counts your false positives, recall counts your false negatives, and tightening one tends to inflate the other.

The F1 score squeezes both into a single number, the harmonic mean of precision and recall:

F1 = 2 * (precision * recall) / (precision + recall)

For the fraud model above that works out to 2 * (0.75 * 0.90) / (0.75 + 0.90) = 0.82. The harmonic mean sits closer to the lower of the two inputs than a plain average would, so F1 only looks good when precision and recall are both decent. It is handy for comparing models, but it hides which error hurts more, so keep the two numbers beside it. To see the trade-off across every threshold at once instead of at one cut-off, you look at a precision-recall curve, or at the ROC curve and its AUC.

Which one should you optimise for?

There is no universal answer: it depends on what a mistake costs. The deciding factor is usually whether a person sees the output or the system acts on it alone.

• A human reviews the output: favour recall. For a fraud alert that lands in an analyst's queue, a false alarm costs a minute of review, but a missed fraud costs real money. You would rather catch everything and let the reviewer discard the false ones, so you accept lower precision to push recall up. The same logic drives anomaly detection and safety screening.

• The system acts automatically: favour precision. If a positive prediction blocks a card, rejects an application, or fires off a warning email with no human in between, every false positive hits a real customer. A wrong flag is expensive and visible, so you keep the threshold high and accept that you will miss some real cases.

Putting rough numbers on it settles the argument faster than any metric: cost out a false positive and a false negative, and the metric you should chase falls out on its own.

What to watch out for with precision and recall

• Define the positive class first. Both scores are measured against the one class you call positive. Swap which class that is and both numbers change, so pin it down before you read them.

• Look at the counts, not only the percentages. Ninety percent recall sounds fine until you learn there were ten real cases and the one you missed was the one that mattered. On a small positive class, a single case swings the number a lot.

• Watch for class imbalance. When positives are rare, plain accuracy can look excellent while the model catches almost none of them. Precision and recall exist precisely for that skew, so read them instead of accuracy on imbalanced data.

• Recall goes by other names. In medicine and statistics it is called sensitivity or the true positive rate, so the same metric can appear under a different label depending on who wrote the report.