Retry policy

What is a retry policy?

A retry policy is the set of rules a system follows when an operation fails: which failures it retries, how many attempts it makes, how long it waits between them, and when it gives up. Every call that leaves your system can fail for a moment: a network blip drops a connection, or an API is briefly overloaded.

Set it too timid and a one-second hiccup fails a whole job. Set it too aggressive and a small outage becomes a large one, as retries pile onto a service that is already struggling. A retry policy belongs wherever a system calls something it does not control: an API client, a data pipeline, or a workflow engine.

Which errors are worth retrying?

The test: could the exact same request succeed a moment later?

• Retry transient faults. Timeouts, dropped or reset connections, 429 Too Many Requests (a rate limit), and 503 Service Unavailable or other 5xx server errors. On a 429 that carries a Retry-After header, wait exactly that long instead of guessing.

• Never retry permanent faults. 400 Bad Request and validation errors, 401 Unauthorized, 403 Forbidden, or 404 Not Found. None of them fixes itself on a second try.

Google Cloud Storage draws the same line, treating 408, 429 and 5xx plus socket timeouts as retryable.

Retries are only safe when the operation is idempotent

This is the precondition, not a footnote. Before you add a single retry, check for idempotence: running the operation twice leaves the system in the same state as running it once. If it is not idempotent, do not retry it until you have made it so.

A timeout is ambiguous. You cannot tell whether the server never received your request, or processed it and only the reply got lost. Retry a non-idempotent create in that state and you can charge a customer twice or raise a duplicate invoice. A payment POST is the classic case: attach an idempotency key and the server recognises the repeat and returns the original result instead of charging again. The same protection matters for a queue consumer under at-least-once delivery, where the broker may deliver the same message more than once.

Budget your retries so they cannot amplify an outage

If a downstream service returns 503 and every client retries five times at once, you multiply the traffic hitting a service that is already over capacity. That is a retry storm.

A retry budget caps it: a maximum number of attempts per request, and a deadline after which you stop and report failure. Google's SRE teams also hold retries below ten percent of all requests, which keeps the added load near 1.1 times normal instead of nearly three times. Pair the budget with exponential backoff and jitter, so waits grow after each attempt and clients do not all retry on the same tick. Retry at only one layer of the stack, because stacked retry loops multiply. For a service failing for a while rather than a moment, add a circuit breaker that pauses requests until it recovers.