Retry storms: why your service can't recover on its own

The thing keeping your service down is usually not the fault that took it down. It's your own clients, retrying. A database hiccups for ten seconds; every layer above it dutifully retries; the retries multiply on the way down; and by the time the original problem clears, the service is being hit with many times its normal traffic by software that is only trying to help. This is a metastable failure — the system stays down after the cause is gone, held there by its own recovery behaviour. It's one of the most common ways a minor incident becomes a long one, and the fix is unglamorous config rather than architecture.

The multiplication is not linear

Most teams reason about retries locally: this call retries three times, so worst case it's three times the work. That intuition is correct for one call and badly wrong for a system. Google's SRE book spells out why, with the arithmetic done:

If the database can't service requests because it's overloaded, and the backend, frontend, and JavaScript layers all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4³) on the database.

Sixty-four. From one person clicking one button, with every individual layer behaving reasonably. Nobody in that stack configured anything crazy — three retries is a sensible-looking default, and it appears three times.

Now scale it. The same chapter walks through the escalation: the retry volume "grows: 100 QPS of retries in the first second leads to 200 QPS, then to 300 QPS, and so on." And it only takes a small overshoot to start the slide — their worked example has a backend with a 10,000 QPS capacity receiving 10,100 QPS, where the failed 100 requests retry, add to the next second's load, fail again, and compound until the backend crashes outright.

That's the shape of the problem. The overload isn't caused by users. It's manufactured by the system's own error handling, and it grows fastest exactly when the service is least able to absorb it.

Why backoff alone doesn't save you

The standard answer is exponential backoff: wait 1 second, then 2, then 4. It's necessary, and on its own it is not sufficient, for a reason that's easy to miss.

Everyone failed at the same moment. If a service returns errors at 14:03:00, every client that was mid-request learns about it simultaneously. They all start their backoff timers at the same instant, and they're all running the same doubling sequence. So they all retry at 14:03:01. Then all of them at 14:03:02. Then 14:03:04.

Exponential backoff spaces out one client's retries. It does nothing about the correlation between clients. You've converted a continuous flood into a series of synchronised tidal waves — which is arguably worse, because each wave hits a service that had just begun to catch its breath.

The fix is randomness. AWS published simulations of this comparing no-jitter backoff against several jittered variants, and their conclusion was that the jittered approaches "cut down work substantially relative to both the no-jitter approaches" — on both total work and completion time. (Their results are presented as graphs rather than a table, so take the shape of the finding rather than precise figures from it.) The AWS authors note how counterintuitive it feels to improve a system by adding randomness. It isn't, once you see that the thing you're fixing is coordination, not timing.

Google's guidance is a single sentence and worth adopting verbatim: "Always use randomized exponential backoff when scheduling retries."

Rejecting the load isn't free either

A tempting middle path is to let the retries come and simply reject the excess quickly. Load shedding is a real technique, but it has a limit people underestimate:

The backend can become overloaded even though the vast majority of its CPU is spent just rejecting requests.

Parsing a request, authenticating it, checking a quota and returning a 429 is cheaper than serving it — but it is not free, and under a 64× amplification the arithmetic stops working. Your service can die at 100% CPU having successfully served nobody. This is why the useful controls live on the client side of the call, not just the server side.

The controls that actually work

1. Cap attempts per request. "Don't retry a given request indefinitely." Three is usually plenty; the second retry rarely succeeds when the first didn't.
2. Add a retry budget per process, not per request. Google's example: "only allow 60 retries per minute in a process, and if the retry budget is exceeded, don't retry; just fail the request." This is the single highest-value control here, because it puts a ceiling on total amplification no matter how many individual calls are failing.
3. Randomise every delay. Full jitter is the simple default: sleep a random duration between zero and the current backoff ceiling.
4. Retry at exactly one layer. This is the discipline that kills the 4³. Pick the layer with enough context to know whether a retry is even sensible, and make every other layer pass the failure through untouched. Retrying in a client library and the service and the gateway is how you get a multiplier you never wrote down.
5. Only retry what's retryable. A 400 or a validation failure will fail identically forever; retrying it is pure amplification with zero chance of success. Timeouts and 503s are worth retrying. Know which is which in your code.
6. Throttle on the client when rejections spike. Google's adaptive throttling has clients reject their own requests locally once the backend's accept rate falls, with a multiplier controlling aggressiveness — they "generally prefer the 2x multiplier," allowing somewhat more traffic through than will be accepted so that clients learn the current state quickly.
7. Make retry rate a first-class metric. If you can't see retries separately from real traffic on a dashboard, you will diagnose this as "mystery traffic spike" every single time. This is precisely the traces-over-logs argument: the aggregate says load is up, the trace says it's the same request four times.

Our opinion

The retry is the most under-designed line of code in most production systems. It gets added during an incident, by someone tired, as a two-line change that makes the immediate symptom go away — and then it is never revisited, because it never fails in a way anyone attributes to it. We've yet to review a system where somebody could tell us, off the top of their head, the maximum number of times one user action can hit the database. That number exists. It's usually a lot larger than people guess, and computing it takes ten minutes.

We'd also push back on the instinct to treat this as an argument against retries. It isn't — transient failures are real, and a system that gives up on the first blip is worse. The problem is never that a system retries; it's that the retries are unbounded, uncoordinated, and duplicated across layers. All three of those are configuration decisions, which is the good news.

One connection worth making: a retry storm rarely announces itself as a retry storm. It shows up as a resource ceiling being hit somewhere downstream — most often a connection pool being exhausted, because 64× the requests need 64× the connections. If your incident timeline reads "database connections maxed out, CPU fine," look upward before you look at the database.

How Ashvara helps

We set retry policy deliberately when we design a system — where the budget lives, which single layer owns the retry, what's classified as retryable, and what the dashboard shows so a storm is legible in the first sixty seconds rather than the fortieth minute.

That's routine DevOps and cloud work joined to how we build backends and APIs. If you've had an outage that outlasted its cause, or one where the graphs looked healthy right up until everything stopped, tell us what happened — this pattern is worth ruling out first.

Sources: Google SRE Book — Addressing Cascading Failures and Handling Overload; AWS Architecture Blog — Exponential Backoff and Jitter. Retry defaults vary widely between SDKs; check what yours does before assuming it does nothing.