Schema registry

What is a schema registry?

A schema registry is a central store for the schemas of the messages that move across a streaming platform. A schema sets out which fields a message carries, their types, and which are required. Instead of every application keeping its own copy, one versioned copy lives in one place.

The idea grew up around Apache Kafka and event-driven architecture, where a producer publishes messages that dozens of consumers read on their own schedules. Nothing stops the producing team from renaming a field or switching a type; a nightly batch surfaces that the next morning, but a live stream carries the bad message to every consumer at once.

The registry checks each version when you register it, before it reaches production. If the change breaks the agreed shape, it rejects the registration and the previous version stays valid, so a producer cannot ship a breaking change into a topic that consumers depend on.

What travels on the wire

What makes a registry practical is what it does not send. When the serializer on the producer registers a schema, the registry hands back a globally unique schema ID. The producer writes a short header on each message, one magic byte and that four-byte ID, then the serialized payload. The full schema never travels with the message.

The consumer reads the ID, fetches the matching schema once, and caches it locally, so later messages with the same ID need no further lookup. Field names and types stay in the registry rather than in every record, which keeps payloads small on a topic carrying millions of messages an hour.

Compatibility modes

Each subject, the named line of schema versions behind a topic, carries a compatibility rule, and the registry tests every new version against it. Confluent Schema Registry ships seven modes. Backward, the default, lets a consumer on the new schema still read data written with the previous one. Forward is the mirror image, full demands both, and none turns the check off. Each mode also has a transitive variant checking every earlier version. The same pattern powers AWS Glue Schema Registry and the Azure Schema Registry in Event Hubs.

Which edits each direction allows, adding an optional field, dropping one, changing a type, belongs to schema evolution, so this entry names the modes and leaves those rules there.

A change the registry rejects

Take a customer-events topic whose subject is set to backward, the default. Version 1 has customer_id, name and email, all required, plus an optional phone.

Adding a required postcode with no default value is rejected. A consumer on the new schema would read an old message that has no postcode, find no value and no default to fall back on, and fail. The registry refuses it, so you see the error while deploying, not in a dead pipeline. Add the field as optional, or give it a default, and it registers cleanly. Catching the break at registration is how a registry keeps schema drift out of a topic.

Schema registry versus data contract

A data contract is the wider agreement between the producer and consumers of a dataset: the schema, plus quality rules, ownership and a change process. A schema registry covers one slice of that, the schema and its compatibility, and enforces it automatically on every message.