Durable execution

What is durable execution?

Durable execution runs a process so that it survives whatever happens to the machines underneath it. The run can take three minutes or three weeks, a server can crash halfway, and you can deploy while it waits. It continues where it stopped instead of starting over or quietly disappearing.

Restate, one of the engines in this category, calls it a programming paradigm that makes ordinary code resilient to crashes, restarts and infrastructure failures. Temporal states the same promise as a guarantee that your application runs to completion.

Three ordinary situations explain why anyone builds this.

• Nothing that waits three days for a manager's approval can live in a variable in memory.

• A payment step times out. You do not know whether the money moved, so the second attempt has to know that step already ran.

• A release goes out while forty processes are mid-run. Every worker restarts, and those forty runs have to still be there.

What sets it apart from a workflow you draw in a designer is that the process stays ordinary code: call a step, wait three days, call the next step. The engine makes that function survive.

How the engine remembers where it was

Every step is written to a durable log before its result comes back to your function. Restate puts the rule directly: every meaningful step, an external API call, a database write, a sleep, a message sent to another service, is recorded to a persistent log before its result is returned. Temporal calls its version the Event History, a complete and durable log of everything that has happened in a workflow execution.

Recovery works by replay, and Microsoft's Durable Task documentation spells it out. When there is more work to do, the orchestrator wakes up and re-executes the entire function from the start to rebuild the local state. Each time the code reaches a step, the framework checks the history, and a step that already produced a result hands that result straight back. At the first thing that has not happened yet, the replay stops and the code runs for real.

Your local variables are never saved. They are rebuilt by running the same code again over recorded answers. That splits the program in two: workflow code, which gets replayed and may only decide, and activities, which touch the outside world once and put their result in the log.

With the state in the engine, four things stop being code you write. A wait measured in days: Temporal says a workflow can sleep for months, and the persisted timer resolves as soon as the worker is back. A wait for a signal from outside, which Azure calls an external event and AWS Step Functions calls a callback with a task token. A retry policy per step. And a full history, which Microsoft notes is what makes reliable compensating actions possible when a later step fails.

The rules that come with replay

Replay is where developers get burned, because it limits what workflow code may do. Microsoft states it flatly: an orchestrator replays multiple times and must produce the same result each time.

So these are out inside workflow code.

• The clock. DateTime.Now gives a different value on every replay. Use the context's own current time.

• Random numbers and fresh GUIDs. Use the context's replay-safe generator, or take the value from a recorded step.

• Direct network and database calls. Microsoft's rule is that orchestrators never make direct HTTP calls of their own, because replay would cause duplicate I/O. Durable Functions gives you a durable HTTP API for the cases where the call itself has to be recorded. Temporal says the same about API calls, database queries and model invocations: they go in activities.

• Sleeping the thread. time.sleep blocks a worker for real. Use the engine's durable timer.

• Environment and static variables. Both can change between the run and a replay two days later. Pass configuration in as input.

Break the rule and the failure is loud. Temporal returns a non-determinism error when a replayed command does not match the event history, and the Durable Task Framework raises a NonDeterministicOrchestrationException, though Microsoft warns it does not catch everything. The rulebook fits in four words: workflow code decides, activities do.

A three-day approval that survives a deploy

Take a purchase order above 5,000 euro that needs a manager's sign-off.

1. Monday 09:12. The run validates the order, reserves the budget in the ERP as reservation R-4417, and sends the approval mail. Three results in the log.

2. Monday 09:13. It reaches the line that waits for an approval or for three days. Nothing is in memory now. The engine holds a timer due Thursday 09:13 and an open wait for an event named approval.

3. Tuesday 14:00. You deploy a release and every worker restarts. The run is untouched, because there was nothing in a process to lose.

4. Wednesday 08:40. The manager clicks approve. A worker replays the run: the validation, R-4417 and the mail step all hand back their recorded results, and the wait now has an answer.

5. Replay reaches the booking, the first thing that never happened, and runs it for real. The approval mail went out exactly once.

Change one detail and it breaks. Had the Tuesday release added a notification step between the budget reservation and the mail, Wednesday's replay would reach a call the history does not contain and fail with a non-determinism error. Microsoft names the causes: changing an activity's name or its input or output type, and adding, removing or reordering calls to activities, timers or external events. Ship that with no plan and you get failed runs, or runs stuck on running forever. The answers are orchestration versioning, a side-by-side deployment on its own task hub, or stopping the in-flight runs on purpose.

Durable execution versus a retry on a normal job

Both answer the same complaint: the job failed, run it again. They differ on one dimension, what is remembered between attempts.

A retry on a normal job remembers nothing. The second attempt starts at line one with empty memory, so everything the first attempt did to the outside world happens again, unless you built the protection: an idempotency key, a table of processed records, a status column you check before every step.

Durable execution remembers every completed step. The retry replays those results and does real work only from the first step that never finished. The mail is not sent twice because the send is not executed twice.

A plain retry fits a short job that is safe to run from the top, such as a nightly load into a staging table you truncate first. Neither approach makes idempotence unnecessary: a step can time out after the far side already committed, and the engine, seeing no result, runs it again.

An agent run has the same shape

An AI agent run is the same problem, which is why the durable execution vendors moved into that market. Restate's engineering write-up puts it plainly: an agent is just regular code, a loop that gathers context, calls tools or a model, works out what to do next and repeats. Every tool call is a remote hop, every user interaction is a pause, every retry risks doing the same work twice, and one network problem can lose everything the run built up.

A model call that takes forty seconds and costs money is the last step you want to repeat because a container was recycled. So the frameworks journal model calls and tool invocations and replay to the last completed step, and hosted agent runtimes describe their sessions the same way. The determinism rule holds here too. A model call is the least predictable thing in the system, which is exactly why it belongs in a recorded step and never in replayed code.

What to watch out for with durable execution

You probably do not need Temporal. Most companies run their processes in Power Automate, Logic Apps or their ERP's workflow module, which have a durable engine underneath already. You want the term so you can name what went wrong when a case disappeared last month, not so you can go shopping.

Know your platform's ceilings. A Power Automate cloud flow has a run duration limit of 30 days from the run's start, pending approvals included, and after that any pending step times out. Run history is kept for 30 days too. So an approval nobody touched for five weeks does not fail loudly: it times out, and a month later the evidence is gone as well.

Treat process code like a database schema. It has live instances inside it, so a change to the step sequence needs a plan the way a column rename does.

The log holds customer data. Step inputs and outputs go to durable storage, so names, amounts and mail bodies land there. Check the retention period and who can read the history.