Retries that make the outage worse: the retry storm problem
Retrying a failed request looks like the obvious fix. Without coordination, it's one of the most common ways to turn a small failure into a full outage.
Retrying a failed operation is, almost always, the first line of defense taught against transient network failures. It's also, without the right coordination, one of the most common ways to turn a small problem into a complete system outage.
How a retry storm forms
Picture a service that starts responding slowly, not fully down, just slow, maybe due to a momentary load spike. Clients depending on it start getting timeouts and, following the reasonable logic of "retry on failure," fire a new request almost immediately. That retry adds more load to exactly the service that was already struggling to keep up, which produces more timeouts, which produces more retries. The mechanism designed to add resilience ends up accelerating the collapse.
This isn't a rare hypothetical. It's the pattern behind a good chunk of the cascading failures seen in real distributed systems: one component degrades a bit, and the collective reaction of its clients finishes it off.
Why "just retry" isn't enough
Three ingredients separate a reasonable retry from a dangerous one:
Exponential backoff. Retrying immediately after a failure is almost always the worst option. Waiting an interval that grows with each failed attempt (1 second, then 2, then 4) gives the system real time to recover before the next hit.
Jitter. If a thousand clients fail at the same time and all use the exact same backoff, they'll all retry at the exact same time, in sync, recreating the same load spike that caused the original problem. Adding randomness to the wait time (jitter) breaks that synchronization and spreads retries out over time.
A real retry limit. Without a ceiling, a client can keep retrying indefinitely against a service that simply isn't going to recover soon, burning its own and everyone else's resources for no benefit.
The ingredient that's almost always missing: the circuit breaker
Backoff and jitter improve the individual behavior of a retry, but they don't solve the underlying problem when a service is genuinely down, not just slow. That's where the circuit breaker pattern comes in: after a certain number of consecutive failures, the client stops trying altogether for a window of time, failing fast instead of continuing to hit a service that isn't going to respond. This protects the failing service (giving it room to recover without extra traffic) and protects the client (it stops burning time and resources on requests almost certain to fail).
The lesson
Naive retry logic optimizes for the happy path: an isolated, momentary failure resolved by a second attempt. Mature design assumes the adverse case: a failure affecting many clients at once, where the uncoordinated collective reaction is itself the next incident. Exponential backoff, jitter, and circuit breakers aren't optional optimizations for large-scale systems. They're the difference between a failure that contains itself and one that amplifies itself.