Checkpointing

What is checkpointing?

Checkpointing is how a long-running job periodically saves its progress to durable storage so that after a crash it can restart from the last known-good point instead of from the beginning.

You meet it most in streaming data pipelines and long jobs on Apache Spark or Apache Flink. A stream never ends, so the job cannot start over from scratch; it has to remember where it was.

What a checkpoint stores

A checkpoint saves two separate things, and recovery needs both.

• The source offsets. How far the job has read from each input. On Apache Kafka this is the committed offset per partition: the last position stored securely, and the exact point a consumer resumes from after a restart.

• The operator state. The running results the job holds while it works: aggregation totals, join buffers, open time windows. A watermark decides when old state can be dropped, so the checkpoint and the watermark travel together.

Take a job counting orders per customer off a Kafka topic. It has read to offset 5000 on partition 0 and holds a running count per customer. The checkpoint records both, so a crash at offset 7000 restarts from 5000 with those counts intact rather than replaying from offset 0.

Checkpointing and exactly-once

Checkpointing alone does not give you exactly-once results. After a crash the job replays everything since the last checkpoint, and whatever already reached the outside world gets produced twice. If that job also writes each customer total to a dashboard table, it checkpoints at offset 5000, writes totals up to 7000, then crashes, and on restart it replays 5000 to 7000 and writes those totals again. The dashboard is now double-counted.

More checkpointing does not fix this; the sink does. If the destination is idempotent or transactional, replaying the same records lands the same final state instead of doubling it. Checkpointing plus an idempotent sink is what people mean by effectively-once, the honest version of exactly-once processing, and it is why idempotence keeps coming up in streaming design.

What to watch out for with checkpointing

The real trap is code changes. A checkpoint belongs to one specific query, and change the query enough and the old checkpoint no longer fits.

In Apache Spark Structured Streaming the rules are explicit. Changing the number or type of input sources is not allowed from an existing checkpoint, because Spark identifies sources by their position in the query plan. Changing a stateful step, such as adding a grouping key or altering the state schema, is not allowed either: the restarted query is likely to fail with unpredictable errors. Adding a filter or changing a rate limit is usually fine.

So a logic change on a live stream often forces a fresh checkpoint and full reprocessing.

Checkpointing versus a database checkpoint

The same word means something else in database engines. A streaming checkpoint saves offsets and operator state so a failed job can resume. A database checkpoint, in an engine like PostgreSQL, flushes all changed data files to disk at a point in the write-ahead log, so crash recovery stays bounded and the log can be trimmed. That is disk housekeeping, not a resume point for a running computation.