Watermark

What is a watermark?

In stream processing, a watermark is the engine's assertion about how far event time has advanced through a stream. Records rarely arrive in the order they happened: a device goes offline, a partition lags behind. The watermark is how the engine says "I now believe I have seen everything up to time T", so it can stop waiting and act.

That one statement drives two decisions: when a time window can close and emit its result, and how late a straggling event may be before it is dropped. Apache Spark Structured Streaming and Apache Flink both build their event-time processing on it, and any pipeline reading streaming data off a source like Apache Kafka has to reason about it.

How the engine sets the watermark

A watermark is not a wall-clock time. It is derived from the data: the highest event time the engine has seen so far, minus a delay you choose:

df.withWatermark("event_time", "10 minutes")

That delay is your bet on how out of order the stream can be. Flink calls the same idea bounded out-of-orderness: a watermark of value T asserts that every event with a timestamp at or before T has already arrived. A larger delay tolerates more stragglers but holds state longer and delays results. A smaller delay closes windows sooner but drops more late data.

A worked example

Take ten-minute tumbling windows with a ten-minute watermark delay. The engine has processed events up to event time 12:14, so the watermark sits at 12:04 (12:14 minus ten minutes).

An event with event time 12:04 arrives at 12:11, late by the clock. Because its event time equals the watermark, it is still counted and updates the 12:00 to 12:10 window. Later the stream reaches event time 12:21, which moves the watermark to 12:11. The 12:00 to 12:10 window is now older than the watermark, so it is finalised and its state is released. A straggler with event time 12:04 arriving later sits below the watermark and is dropped: its window is already gone.

Watermark versus high-water mark

The word also appears in batch work, where a high-water mark is the stored position an incremental load resumes from: the highest updated_at or largest id from last run. Both ideas share a boundary that only moves forward, but the mechanism is different.

A streaming watermark is live event-time bookkeeping the engine recomputes as data flows, and it decides drops and window closures inside a running job. A high-water mark is a durable value written to a control table between runs, so the next Incremental load knows where to start. The streaming watermark is itself part of the job state that Checkpointing persists, so a restarted job carries on without replaying the whole stream.

What to watch out for with watermarks

• The watermark only moves with data. If a source goes idle or one partition stalls, the watermark freezes and windows never close. A single slow partition holds back the whole job, because the watermark is the minimum across inputs.

• Dropped is dropped. Once an event falls below the watermark, most engines discard it without warning. If you still need those records, capture them separately, for example Flink can route late events to a side output instead of losing them.