Schema evolution

What is schema evolution?

Schema evolution is how you change the shape of a dataset over time, adding a column, renaming one, or changing a type, without breaking the systems that read and write it. The shape in question is the schema: which columns or fields exist, what they are called, and what type of value each one holds. When only the values change, a customer moves or an order changes status, the structure holds. Schema evolution is when the structure itself changes, the case that quietly breaks pipelines.

It shows up in two places. In table formats like Delta Lake and Apache Iceberg, the schema describes a table that queries and reports depend on. In message streams on Apache Kafka, it describes the events that producers write and consumers read. Either way, a change to the shape has to reach every reader and writer without stranding one of them.

Safe changes and breaking changes

Not every change carries the same risk. What matters is whether a reader written against the old schema can still make sense of data written under the new one.

Changes that are usually safe:

• Adding a nullable or optional column. Existing readers ignore a column they do not know about, and readers that expect it fall back to the default or null when older records lack it.

• Widening a type. Moving an integer to a larger integer, or a float to a double, keeps every existing value representable. The Avro specification allows exactly these promotions: int to long, float, or double; long to float or double; float to double.

Changes that break readers:

• Renaming a column. A reader that matches by name still asks for the old name and finds nothing.

• Dropping a column. Any query or transformation that referenced it stops.

• Narrowing a type. Going from long to int, or from a string to a number, can truncate values or fail outright, sometimes without an error.

The silent ones are the dangerous ones. A rename or a drop usually throws an error you notice that night, but a narrowing type passes the load and only surfaces later as a wrong number in a report.

Backward, forward and full compatibility

When schemas travel over a stream, a schema registry decides in advance which changes are allowed. On Kafka, the Schema Registry from Confluent is the usual one: you register each schema version and set a compatibility rule, and a change that breaks the rule is rejected before it reaches anyone. Three directions matter here, and they are easy to mix up.

Backward compatibility
A consumer using the new schema can read data written with the previous schema. You may delete fields and add optional fields. Because the reader moves first, you upgrade consumers before producers. This is the default in Schema Registry.

Forward compatibility
A consumer still on the old schema can read data written with the new schema. You may add fields and delete optional fields. Here the writer moves first, so you upgrade producers before consumers.

Full compatibility
Both directions hold at once. You may add optional fields and delete optional fields, and producers and consumers can be upgraded in any order.

A worked case: your order events gain an optional discount_code field with a default. Under backward compatibility you upgrade the consumers first; when one meets an older message with no discount code, it reads the default. Add that same field as required, with no default, and the registry rejects it: a new consumer would have nothing to fall back on for the old messages.

How Iceberg avoids the rename problem

A rename is dangerous mainly because many formats match columns by name. If a reader looks up a column called qty and you rename it to quantity, the read comes back empty, and a later column that reuses the old name can even resurface stale data.

Apache Iceberg removes that trap by tracking every column with a unique ID rather than its name or position. The ID lives in the table metadata and is written into each Parquet data file, and reads match on the ID. Renaming a column changes a name in the metadata and nothing else; the data files are left alone. Iceberg treats add, drop, rename, widen and reorder all as metadata-only operations, so none of them rewrite data.

Because IDs are never reused, a newly added column never reads values left behind by an old one, and a drop or a reorder cannot shift values onto the wrong name. Delta Lake reaches the same rename safety through its column mapping feature, which separates a column's logical name from its physical name in the files.

Schema evolution versus schema drift

Schema evolution and schema drift describe the same event, a schema that changed, but they split on one dimension: intent. Evolution is the change you plan: you agree on the new shape, pick a compatible direction, and roll it out so readers and writers stay in step. Drift is the change nobody agreed to, where a source silently renames a field or switches a type and your pipeline finds out by failing or, worse, by producing a wrong figure.

What keeps drift from becoming your default is a data contract, where the source team commits to a schema and a change procedure, plus monitoring that flags a rename or a type switch before a report shows it. Evolution is the version you organise; drift is the version that happens to you.