Overfitting and underfitting

What is overfitting and underfitting?

Overfitting and underfitting are the two ways a machine learning model can fit its training data badly. Overfitting is when the model learns the training data too closely, including its accidental quirks and mistakes, and then stumbles on new data. Underfitting is the opposite: the model is too simple to catch the real pattern, so it does poorly even on the data it trained on.

A model that has learned well sits between the two. It picks up the genuine pattern and ignores the accidental detail, so it holds up on data it has never seen. That ability to perform on fresh data is called generalisation, and it is the whole point of training a model.

Picture two students preparing for an exam. One memorises last year's answers word for word and is lost the moment the questions change: that is overfitting. The other barely skims the material and learns nothing specific, so they struggle on any question: that is underfitting. You want the student who understood the subject.

How to spot each one

The tell is the gap between how the model scores on its training data and how it scores on data held back from training. You get that held-back data from a train-test split: you set part of the data aside and only measure the model against it at the end, so the score reflects genuinely new cases.

• Underfitting. Poor on the training data and poor on the test data. The model is missing the pattern everywhere.

• Overfitting. Strong on the training data but noticeably worse on the test data. That widening gap is the signature.

• Good fit. Solid on both, with only a small gap between them.

The same split shows up in the language of bias and variance. An underfit model has high bias: rigid assumptions that are consistently off. An overfit model has high variance: it swings wildly depending on the exact training data it saw. You cannot drive both to zero at once, so tuning a model is largely about balancing the two.

Why models overfit and underfit

It comes down to how much flexibility the model has, compared with how much data and real signal it has to learn from.

Underfitting shows up when the model is too simple for the job: a straight line trying to follow a curved trend, too few input features, training cut short, or a regularisation setting turned up so high it flattens the model. Overfitting shows up at the other extreme: a model flexible enough to memorise, trained for too long, or handed more features than the data can support. With enough freedom and too little data, the model starts fitting the noise instead of the pattern.

How to catch and fix them

A handful of habits keep a model in the healthy middle.

• Cross-validation. Rather than trusting a single train-test split, rotate the held-out portion across several folds and average the results. A model that only looked good on one lucky split gets exposed.

• Regularisation. Add a penalty for complexity so the model prefers simpler solutions. L1 and L2 penalties on the weights are the common ones, and early stopping halts training the moment the held-out score stops improving.

• More and better data. Overfitting eases when the model has more examples to generalise from, and cleaner labels help too.

• Adjust the complexity. Underfitting calls for a more capable model or richer features; overfitting calls for a simpler one or fewer features.

Overfitting versus model drift

Both leave you with bad predictions on real data, but they happen at different moments. Overfitting is a training-time flaw: the model was never going to generalise, and you would have caught it on the test set before release. Model drift comes later. A model that worked fine slowly gets less accurate because the world it predicts has changed: prices move, customer behaviour shifts, a new product launches. Overfitting is fixed before you ship. Drift is watched for after you ship.

What to watch out for with overfitting and underfitting

Data leakage flatters the score. If information from the test set sneaks into training, the model looks great in testing and disappoints in production. Keep the split clean and decide it before you touch the data.

A tiny test set lies. Judging generalisation on a handful of held-out rows gives a noisy, unreliable number. Keep the test set large enough to trust.

Chasing training accuracy is the trap. A model that scores near-perfect on its own training data is often overfit, not excellent. The held-out score is the one that counts.

You can over-correct. Pile on too much regularisation or strip out too many features to fight overfitting, and you tip straight into underfitting. The goal is balance, not the extreme.