Schema drift

What is schema drift?

Schema drift happens when the structure of a data source changes over time. A column is added, removed, renamed, reordered, or changes data type. The data still arrives, but it no longer has the shape downstream systems expect.

A pipeline, transformation, dashboard, or model that depends on the old schema can break. Worse, it can keep running and produce the wrong result.

Schema drift is common in SaaS exports, APIs, CSV files, operational databases, event streams, and partner feeds. The source team changes the structure for a good reason, but the consumers find out too late.

How schema drift happens

Application changes. A CRM, ERP, webshop, or SaaS product adds a field, renames one, or splits a field into two.

Database changes. A team changes a source table, modifies a type, drops a column, or introduces nullable values where the pipeline expected mandatory ones.

API changes. A response field is renamed, nested differently, deprecated, or moved to a new endpoint.

File changes. A CSV export gains a new column in the middle, changes the date format, or switches decimal separators.

Event changes. A producer changes the shape of messages on Kafka, Event Hubs, or another stream without consumers being ready.

Why schema drift is dangerous

The best failure is loud. A pipeline expects customer_id, the column is gone, the run fails, and someone gets an alert.

The worse failure is quiet. A renamed column lands as a new empty field. A price that used to be euros arrives in cents. A column-order based import reads the customer name where it expected the date. The dashboard still refreshes, but the numbers are wrong.

Quiet schema drift is harder to catch because everything looks operational. The run is green. The data is there. Only the meaning shifted.

Example: a changed CSV export

A webshop sends a nightly CSV with these columns:

order_id, order_date, customer_id, amount_ex_vat, vat_amount

Your transformation calculates total amount as amount_ex_vat plus vat_amount.

After a webshop update, the export changes. The VAT field is renamed to tax_amount, and a discount column appears before it. A name-based pipeline may suddenly produce null VAT. A position-based pipeline may read the wrong column entirely. Either way, the report can understate revenue without anyone changing the report itself.

How to defend against schema drift

Compare incoming and expected schemas. At each run, check names, types, nullability, and required columns. Fail fast on breaking changes.

Use data contracts. Agree what the source promises to deliver, which changes are allowed, and how breaking changes are announced or blocked.

Use a schema registry for streams. In Kafka-style event systems, a schema registry can validate new message versions before consumers receive them.

Test important tables. Data quality tests should check row counts, nulls, accepted values, type expectations, and business rules.

Alert on change. Data observability tools can detect new, missing, or changed columns and notify the owner before users notice.

Handle new columns deliberately. Some tools can automatically include new columns, block all new data, or allow only new columns on existing tables. Choose the policy per source instead of letting it happen by accident.

Schema drift, data drift, and model drift

Schema drift is a structure change. The columns, types, or message shape changed.

Data drift is a value-distribution change. The columns are the same, but the values behave differently: fewer orders, different customer ages, new countries, more nulls.

Model drift is a performance change in a machine learning model. The model becomes less accurate because the world, behaviour, or input data changed.

They can be related. Schema drift can cause data drift by moving values into the wrong fields. Data drift can later cause model drift. But the monitoring signals are different.

What to watch out for with schema drift

Position-based parsing is fragile. Match columns by name where possible. A new column in the middle should not shift every field after it.

Automatic adoption can spread bad changes. Accepting every new column sounds flexible, but it may publish unreviewed fields into the warehouse.

Blocking everything can slow teams down. Some new columns are harmless. Classify changes as allowed, warning, or breaking.

Ownership is the real control. Every source and downstream model needs a named owner who receives schema-change alerts and can decide what to do.