Apache Kafka

What is Apache Kafka?

Apache Kafka is an open-source event streaming platform. It captures streams of events, stores them durably, and lets many systems read those events independently. A source application writes to Kafka once; analytics, microservices, search, monitoring, and data pipelines can each consume the same stream at their own pace.

The core idea is a log. New events are appended to the end. They are not removed just because one consumer has read them. As long as the retention period keeps them, another consumer can read the same events later, replay from an earlier point, or catch up after being offline.

That makes Kafka different from a simple message queue. It is not just a hand-off between sender and receiver. It is a durable event backbone for systems that need to react to what happened and, sometimes, replay what happened.

The main Kafka concepts

Event
An event is a record that something happened: an order was placed, a payment cleared, a sensor reported a value, a row changed in a database. In Kafka, an event usually has a key, a value, a timestamp, and optional headers.

Topic
A topic is a named stream of events, such as orders, payments, or inventory-updates. Producers write events to a topic. Consumers subscribe to a topic.

Partition
A topic is split into partitions. Partitions let Kafka scale because different brokers and consumers can handle different parts of the stream. Ordering is guaranteed within a single partition, not across the whole topic. Events with the same key usually go to the same partition, so all events for one customer or order stay in order.

Offset
An offset is the position of an event inside a partition. A consumer records which offsets it has processed. If it needs to replay an hour of data, it can move the offset back and read again.

Broker
A broker is a server in the Kafka cluster. Brokers store partitions and serve reads and writes. A production cluster usually has multiple brokers for scale and fault tolerance.

Consumer group
A consumer group lets several consumers share the work. Kafka assigns partitions to members of the group so each event is processed by one member of that group. A second application uses a different group and gets its own independent read of the same topic.

Where Kafka fits in a data stack

Kafka often sits between operational systems and the platforms that need their data.

Change Data Capture
A CDC tool can read the transaction log of a database and publish every insert, update, and delete to Kafka. Downstream systems then update a lakehouse, data warehouse, search index, or cache from the same stream.

Event-driven microservices
Instead of one service calling five others directly, it writes an event. Other services react when they are ready. The order service publishes order-created, the inventory service reserves stock, the invoicing service prepares billing, and the analytics layer records the event.

Streaming ingestion
Spark, Flink, or a cloud streaming service can read from Kafka and turn raw events into tables. A lakehouse might keep both the raw event log and a cleaned current-state table for reporting.

Replay and recovery
Because events remain available for a retention period, a new consumer can start from the beginning of a topic, or an existing consumer can replay data after a bug fix.

Delivery guarantees

Kafka systems are usually designed around delivery guarantees. The exact guarantee depends on producer settings, consumer behaviour, transactions, and the target system.

At-least-once means an event should not be lost, but it may be processed more than once after a retry or restart. This is common and practical. Consumers must be idempotent: processing the same event twice should not create a wrong result.

At-most-once means an event is never processed twice, but it may be lost if a failure happens at the wrong moment. That is rarely acceptable for financial or operational data.

Exactly-once is possible in Kafka for specific patterns, using idempotent producers and transactions, but it is not a magic setting for the whole architecture. The sink still needs to participate correctly. A database write, API call, or warehouse merge can reintroduce duplicates if it is not designed carefully.

Kafka or a managed event service?

Kafka is powerful, but running a cluster is operational work: brokers, storage, partitions, replication, upgrades, security, schemas, lag monitoring, and incident response.

For many teams, a managed service is the better first answer. Azure Event Hubs supports the Kafka protocol for many producer and consumer clients, so existing Kafka applications can often connect with configuration changes rather than a rewrite. Microsoft Fabric eventstreams can also capture, transform, and route real-time events inside Fabric, which is useful when the destination is already a lakehouse, eventhouse, or Power BI scenario.

Self-managed Kafka makes sense when you need Kafka-specific control, a large connector ecosystem, custom topology, or portability across clouds. Managed streaming makes sense when the business problem is real-time ingestion, not operating Kafka itself.

What to watch out for with Kafka

Partition count is hard to change casually
Partitions drive scale and ordering. Too few limit throughput. Too many increase overhead. Changing them later can affect ordering assumptions.

Lag needs monitoring
A consumer can silently fall behind. Monitor the delay between the latest event and the offsets your consumers have processed.

Events need schemas
A stream without a schema contract becomes difficult to evolve. A Schema Registry or equivalent contract helps producers and consumers change safely.

Retention is not backup
Kafka can keep events for days, weeks, or longer, but it is still an event log with configured retention. Long-term history usually belongs in object storage or a lakehouse as well.

Not every queue problem needs Kafka
For a few low-volume background jobs, a simpler queue is easier. Kafka pays off when volume, replay, fan-out, or streaming semantics matter.