Data drift

What is data drift?

Data drift is a change in the statistical distribution of the input data a model or report was built on, while the structure stays the same. The columns, the data types and the pipeline are untouched. What moves is the mix of values inside those columns and how they behave.

Some of this is normal. Customers buy differently, new products launch, seasons turn, and the data follows the real world. Drift turns into a problem when a machine learning model or a dashboard was tuned on last year's patterns and nobody notices the ground has moved.

In the research literature this is called covariate shift, a term from a 2000 paper by Hidetoshi Shimodaira: the distribution of the inputs changes while the link between an input and its outcome stays the same. It is a common reason predictions lose accuracy long before anything visibly breaks.

Data drift, concept drift, model drift and schema drift

These four terms get used interchangeably and mean different things. Telling them apart is the whole reason you monitor them.

• Data drift is a change in the inputs. The distribution of the feature values shifts, so the model sees a different mix of data than it trained on. This is covariate shift.

• Concept drift is a change in the relationship between the inputs and the target. The inputs can look identical, but the same input now maps to a different answer.

• Model drift is the result you feel: the model gets less accurate. Data drift and concept drift are two of the reasons behind it, but they are causes, not the decay itself.

• Schema drift is a change in structure, not distribution. A column is renamed, retyped or dropped. That is a different failure with a different fix.

Data drift and concept drift are the pair people confuse most. One changes what goes into the model; the other changes what those inputs are worth.

A worked example

Take a pricing model at an insurer that estimates the expected claims cost of each applicant from features like age, region and vehicle type. A new online channel starts bringing in younger applicants, so the age distribution shifts down. This is data drift. If a thirty year old still carries the same risk a thirty year old always did, the model is still right for each person and only the mix has changed, so accuracy can hold even though the inputs drifted.

Contrast that with a new law that caps payouts. The same applicant, same age, same region, now produces a different claims cost than before. The inputs did not move, their meaning did, and that is concept drift, which almost always drags accuracy down. Whichever effect shows up when you compare predictions against real outcomes is what you measure as model drift.

How you detect it

You cannot eyeball drift across hundreds of features. Detection compares two windows of data: a baseline, often the training set or a known-good period, and the current production data, and measures how far each feature has moved.

The test depends on the feature type. Evidently, an open-source monitoring library, picks a two-sample Kolmogorov-Smirnov test for numeric features, a chi-squared test for categorical ones, and a proportion test for binary flags. Azure Machine Learning builds the same check into its model monitoring: it compares a baseline against recent production data and raises a per-feature drift signal next to data quality and prediction drift signals, scoring numeric features with the Wasserstein distance. Either way, the owner gets alerted once a feature that drives a real decision moves past a threshold.

Why a p-value is not enough

Statistical tests have a trap on large data. Say a fraud model scores 200,000 transactions a day and you run a Kolmogorov-Smirnov test on transaction_amount against the training baseline. At that volume the test returns a p-value under 0.05 almost every day, even when the average has moved by less than a euro. The test is right that the samples differ, but useless as an alarm: it fires on differences too small to change any decision.

Evidently's own testing found the Kolmogorov-Smirnov test flags a shift as small as 0.5 percent once a sample passes roughly 100,000 rows, so its library switches away from p-value tests above 1,000 observations and scores distance instead, with a default threshold of 0.1.

The fix is an effect-size measure with a threshold you choose. The population stability index, or PSI, bins a feature and adds up how much each bin's share of the data has moved. A common rule of thumb from credit scoring reads a PSI below 0.1 as no meaningful change, 0.1 to 0.25 as a moderate shift, and above 0.25 as a large one. The question you want to answer is not whether the distributions differ, but whether they differ enough to act on.

What to watch out for with data drift

Not every drift is a defect. A shift can be a real business change, a data quality problem upstream, or plain seasonality. Work out which before you retrain anything. This is where data quality checks and data observability earn their keep.

Drift on an input is a warning, not a verdict. Confirm it against outcomes. Data drift only turns into model drift when it moves a feature the model actually leans on; a feature the model barely uses is something you can log and leave alone.

Retraining is not the only answer. Sometimes you recalibrate, sometimes you fix the source, sometimes you redraw a segment or a KPI. When you do retrain, you run your feature engineering again on fresh data and record each attempt in experiment tracking, so you can tell whether the new model is genuinely better.