Confusion matrix

What is a confusion matrix?

A confusion matrix is a small table that sorts every prediction a classification model made by what the model said against what was actually true. For a yes/no model it has four cells, each holding a count. Together they show how often the model was right and, more usefully, which mistakes it made.

It applies to any supervised learning classifier: fraud or no fraud, spam or inbox, churn or stay, defect or pass. You need labelled test data, rows where the real answer is known, so each prediction can be scored against the truth.

The name is unfortunate, but the idea is plain: where does the model confuse one class for another? A single accuracy figure cannot show that; the matrix can.

The four cells

Take a model that flags card transactions as fraud, and call fraud the positive class. Every scored transaction lands in one of four cells.

• True positive. The model flags a transaction as fraud and it really was fraud. The alert was right and the loss was stopped.

• False positive. The model flags a genuine purchase as fraud. The customer's card is declined for nothing.

• False negative. The model waves a fraudulent transaction through as normal. The fraud is missed and the money is gone.

• True negative. The model lets a genuine purchase through as normal, which is correct. Usually the biggest cell and the least interesting.

How scikit-learn orders the cells

Most textbooks put the true positive in the top-left corner. scikit-learn, the standard Python machine learning library, does not. Its confusion_matrix function sorts the class labels in ascending order, so with 0 for negative and 1 for positive, the top-left cell holds the true negatives and the true positives fall in the bottom-right.

The documentation is explicit: the count of true negatives is cell (0, 0), false negatives is (1, 0), true positives is (1, 1) and false positives is (0, 1). Read a scikit-learn matrix as if it used the textbook layout and you will swap your errors. The safer habit is to unpack the counts by name rather than by position.

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

That line returns the counts in the order true negative, false positive, false negative, true positive.

Metrics you read off the matrix

The four counts feed the numbers people report. Write TP, FP, FN and TN for the cells.

• Accuracy is the share of predictions that were correct: (TP + TN) / (TP + TN + FP + FN). Fine when the classes are roughly balanced, misleading when they are not.

• Precision is how often a positive prediction was right: TP / (TP + FP). When the model cries fraud, how much can you trust it?

• Recall, also called sensitivity, is how much of the real positive class the model caught: TP / (TP + FN). Of all the fraud, how much did you find?

• Specificity, the true negative rate, is the mirror image for the negative class: TN / (TN + FP).

Precision and recall pull against each other, and plotting that trade-off across thresholds gives the ROC curve and its summary figure, the AUC. Both are topics in their own right.

The cost of the two errors

The real reason to look past accuracy is that a false positive and a false negative rarely hurt the same amount, and the matrix keeps them apart so you can price them separately.

In fraud, a false negative is settled loss while a false positive is a few minutes of review and an annoyed customer, so you tune the model to miss as little fraud as possible even when that means more false alarms. Flip it for a cancer screen, where a false negative is a missed tumour, far worse than a false positive that a second test clears. A spam filter runs the other way: a false positive, a real email dropped into junk, hurts more than the odd spam that slips through. Same matrix, opposite priorities, and only the split counts let you argue it with numbers.

A worked example

Run a fraud model over 1,000 transactions, of which 100 are truly fraud and 900 are genuine. The model produces these four counts:

• True positives: 80. Fraud that was flagged.

• False negatives: 20. Fraud that slipped through.

• False positives: 90. Genuine transactions wrongly flagged.

• True negatives: 810. Genuine transactions correctly cleared.

The two fraud cells add up to 100 and the two genuine cells to 900, matching the split we started with. Now read the metrics off those counts:

• Accuracy = (80 + 810) / 1000 = 890 / 1000 = 0.89, or 89 percent.

• Precision = 80 / (80 + 90) = 80 / 170 = 0.47, about 47 percent. Fewer than half the fraud alerts are real fraud.

• Recall = 80 / (80 + 20) = 80 / 100 = 0.80, or 80 percent. The model catches four fraud cases in every five.

• Specificity = 810 / (810 + 90) = 810 / 900 = 0.90, or 90 percent.

The lesson hides inside the accuracy. A lazy model that labels every transaction genuine would score 900 / 1000, a full 90 percent, beating the real model while catching no fraud whatever. That gap is what class imbalance does to accuracy, and it is why anomaly detection and fraud teams read the whole matrix instead of one headline number. The same check on fresh data catches model drift, when yesterday's strong recall quietly decays.