Class imbalance

What is class imbalance?

Class imbalance is when one label in a machine learning classification dataset vastly outnumbers the others. In fraud detection almost every transaction is legitimate. In churn most customers stay. On a production line most parts pass. For a rare disease most patients test negative.

The rare class is usually the one you actually want to find. Class imbalance turns up in anomaly detection and any supervised learning problem where the interesting event is rare, and it breaks how most people first judge a model.

Why accuracy misleads

Take 10,000 transactions where 1 percent, so 100, are fraud and the other 9,900 are legitimate. A model that predicts "not fraud" every time gets all 9,900 legitimate cases right and all 100 fraud cases wrong. Its accuracy is 9,900 divided by 10,000, or 99 percent, and it catches zero fraud. The score looks excellent while the model is useless.

Now a model that works. It flags 200 transactions: 80 are real fraud, 120 are false alarms, and it misses the other 20 fraud cases. It catches 80 of the 100 fraud cases, yet its accuracy is 80 plus 9,780, divided by 10,000, or 98.6 percent, lower than the do-nothing model. When one class is rare, accuracy rewards the wrong behaviour.

What to measure instead

Two numbers describe the model that matters. Precision is the share of flagged cases that were really fraud, here 80 out of 200, or 40 percent. Recall is the share of real fraud you caught, here 80 out of 100, or 80 percent. They trade off: lower the threshold and recall rises while precision falls. Together they say far more than accuracy on imbalanced data.

To pick a threshold, read the precision-recall curve rather than the ROC curve and AUC. When positives are rare the ROC curve flatters a model: its false-positive rate divides by the huge pool of negatives, so many false alarms barely move it. The precision-recall curve keeps the rare positives in view. Scikit-learn makes the same point: precision and recall are the more useful measure when classes are very imbalanced.

How to rebalance the classes

Three honest remedies, roughly in priority order.

• Class weights. Tell the algorithm to penalise mistakes on the rare class more heavily. In scikit-learn, class_weight='balanced' sets each weight inversely proportional to class frequency, so the 1 percent class carries more weight. No data is copied or invented.

• Resampling. Oversample the minority class or undersample the majority so the training set is more even. This is plain data sampling, and the imbalanced-learn library ships a RandomOverSampler and a RandomUnderSampler for it.

• SMOTE. Instead of duplicating rows, SMOTE builds synthetic data: it places a new point on the line between a minority example and one of its nearest neighbours. Introduced by Chawla and colleagues in 2002, it fills in the minority region rather than copying records.

One rule sits above all three: resample or apply SMOTE only inside the training fold, never before the split. Oversample first and split afterwards, and near-identical copies of the same row land in both training and test, so your scores are inflated by data leakage. Fit any resampling on the training data alone, then evaluate on untouched data.