Data leakage

What is data leakage?

In machine learning, data leakage is when information from outside the training set, often from the future or from the target itself, sneaks into the model during training. The result is a model that scores brilliantly in validation and then falls apart on real data in production.

This is not the security sense of the word. Nothing has been stolen or exposed. The leak sits inside your own project: the training data quietly contains hints about the outcome that will not be available when you actually make a prediction. Leakage is a supervised learning problem: you train on labelled examples to predict a target, and one of those labels has slipped into the inputs.

It is like studying for an exam with the answer key on the desk. You ace the practice test, then fail the real one. Leakage is dangerous because the warning signs look like success: high accuracy and a clean validation curve, right up until the model meets data it has genuinely never seen.

The common forms of leakage

Target leakage

Target leakage is a feature that is only known after the outcome it is meant to predict. Picture a model that flags customers likely to default on an invoice, with one column holding the number of collection letters sent. That looks strongly predictive, because customers who default get lots of letters. But the letters are a consequence of the default, not something you can see beforehand: at prediction time, none have gone out yet. The feature is a stand-in for the label, so the model learns to read the answer instead of the situation.

Train-test contamination

This is leakage during preprocessing. Scaling, imputing missing values, encoding categories and selecting features all learn something from the data they see. If you fit a scaler or an imputer on the whole dataset before you split it, statistics from the test rows bleed into training. The scikit-learn documentation shows how sharp this is: fit a feature selector on the full dataset, then split, and a model trained on completely random labels still reaches about 76 percent accuracy where honest chance is 50 percent. Automated feature engineering is especially prone to this, because it computes across every row it is handed.

Temporal leakage

Temporal leakage is the time-series version. When your data has an order in time, a random split scatters future rows into training and past rows into test, so the model gets to look ahead. A sales forecast built this way learns from next month to predict last month. The honest approach is to split by time: train on the earlier period and test on the later one, the way the model will actually run.

Group leakage

Group leakage is when rows that belong together land on both sides of the split. If you have several visits per patient, several orders per customer or several readings per device, a random split can put some of a patient's rows in training and the rest in test. The model then recognises the individual instead of learning the pattern, and the test score flatters it. Every record from the same group has to stay together in one fold.

How to prevent leakage

Every fix comes down to one habit: only let the model learn from what it would genuinely have at the moment of prediction.

1. Split first. Do the train-test split before you scale, impute, encode or select features. Nothing that learns from data should see the test rows.

2. Fit preprocessing inside the fold. Wrap your transformers and model in a single pipeline so each cross-validation fold fits its own scaler and imputer on its own training rows.

3. Split by time for time-series data. Train on the past and test on the future, never a random shuffle.

4. Split by group when records cluster. Keep each patient, customer or device wholly inside one fold.

5. Check every feature for hindsight. For each column, ask whether its value is known at prediction time. If it is filled in only after the outcome, drop it or rebuild it as a point-in-time version.

A pipeline makes the first two rules automatic, because the scaler is only ever fitted on the training slice:

X_train, X_test, y_train, y_test = train_test_split(X, y)
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)   # the scaler learns from training rows only

For the hindsight check, a feature store with point-in-time joins can serve each feature as it looked on the date of the event rather than as it looks today, which stops future values slipping in. Building this discipline into your data pipeline and mlops process is cheaper than debugging a model that already shipped.

Data leakage versus overfitting

Leakage and overfitting both make a model look better in testing than it really is, but they are different faults. Overfitting is a model that memorises the noise in its training data and fails to generalise. You fight it with more data, a simpler model or regularisation, and it usually shows up as a gap between a high training score and a lower validation score.

Leakage is a flaw in how the data was put together: the information was never fair to use. Its tell is the opposite, a validation score that looks too good rather than a train-validation gap. A simple model can score almost perfectly under leakage, because the answer is sitting in the input. Worth separating from both is model drift, which is a model going stale as the world changes after deployment. Drift happens later, leakage is baked in before the model ever ships.