Backpressure

What is backpressure?

Backpressure is what happens when the part of a system that receives data cannot keep up with the part that sends it, and gets a way to say so. The slow side signals that it is full, and the fast side responds by slowing down, holding data in a buffer, or dropping some of it on purpose. The name comes from plumbing: push water through a pipe faster than the far end can drain, and pressure builds back up towards the source.

The problem it solves is simple to picture. A producer emits 10,000 records a second. The consumer that reads them can handle 3,000 a second. Without any flow control, the 7,000 record gap has to go somewhere every second. It piles up in memory until the consumer runs out and crashes, and often takes the whole pipeline with it. Backpressure is the agreement that keeps that gap from growing without limit.

This shows up anywhere data moves between a faster part and a slower part: streaming data platforms, message queue systems, network sockets, and the reactive libraries inside modern applications. It is a normal, healthy signal, not a failure. A pipeline that can apply backpressure degrades gracefully under load instead of falling over.

What happens without it

When a system has no way to push back, load has only bad places to go.

The first is memory. Unread data collects in an in-memory queue that grows until the process is killed. The second is latency: even before it crashes, a service holding a huge backlog answers every request slowly, because each new item waits behind everything already queued. The third is data loss, and it is the worst kind, because it is silent. A buffer overflows, records are quietly discarded, and nobody notices until a report comes up short.

The point of backpressure is to turn that hidden, catastrophic failure into a visible, controlled one. A pipeline that slows down under pressure is telling you something useful. A pipeline that drops data without saying so is lying to you.

How systems apply it

There are a few honest responses to more data than you can handle, and a real system usually combines them.

• Slow the producer. The cleanest option: tell the source to send less until the backlog clears. This works only when the producer can actually be told to wait, which is not always the case.

• Buffer. Hold the excess in a queue and work it off. This smooths out short spikes well, but a buffer is a delay, not a cure. If the overload lasts, the buffer fills and you are back to choosing between blocking and dropping.

• Drop. Discard data on purpose when it is safe to do so. For a live dashboard showing an approximate metric, skipping the odd reading does no harm. For payments, dropping is not an option.

The deeper distinction is push versus pull. In a push model the producer sends whenever it wants, so backpressure has to be added on top. In a pull model the consumer asks for the next batch only when it is ready, so flow control is built in: the consumer that never asks simply never receives. The Reactive Streams specification, which underlies reactive libraries across the Java and .NET worlds, formalises this. A subscriber calls request(n) to state how many items it can take, and the publisher is not allowed to send more than that outstanding demand. TCP has done the same thing for decades: the receiver advertises a window telling the sender how much it may transmit before waiting for an acknowledgement.

Backpressure in Apache Kafka

Apache Kafka handles this gracefully because it is pull-based by design. Producers write records to a durable, ordered log, and consumers fetch from it at their own pace. If a consumer is slow, records are not lost and memory does not blow up: they simply stay on disk in the log, and the consumer's position falls further behind the newest record. That distance is called consumer lag, and it is the main signal that backpressure is building.

Because the log holds the data safely, the consumer's own job is to avoid biting off more than it can chew. The Kafka client lets a consumer pause and resume specific partitions, so an application that is falling behind can stop pulling new records while it finishes what it has, then resume. Watching consumer lag over time tells you whether the consumer is keeping up, catching up, or losing ground. A lag that only ever grows means the consumer is permanently too slow and needs more instances or faster processing, not a bigger buffer.

What to watch out for with backpressure

A buffer is not a solution. Adding memory or a bigger queue buys time, nothing more. If the consumer is structurally slower than the producer, every buffer eventually fills. The fix is more consumer capacity or less incoming work.

Silent dropping erodes trust. If your overload strategy is to discard data, that decision has to be visible: counted, logged, and understood by whoever reads the output. Dropped records that nobody knows about turn into wrong numbers that nobody can explain.

Backpressure moves the bottleneck, it does not remove it. Slowing a fast producer can push the pressure one step further upstream, to whatever feeds that producer. Trace the whole chain rather than treating each hop on its own.

Not everything can wait. Slowing the producer assumes the source will tolerate being told to wait. A sensor, a market feed, or an external partner may keep sending regardless, which forces you back to buffering or dropping. Know which of your sources can be asked to slow down before you design around the assumption that they can.