Use Exponential Backoff

Why the Wait Should Grow

Lesson 143 decided when to retry. This lesson decides how long to wait before each retry, and the answer is a pattern called exponential backoff: the wait starts short and doubles with every attempt, up to a ceiling. The pattern exists for a reason that is easy to miss. When a service is struggling, many clients retry at the same moment, and their retries arrive together, which makes the overload worse. Waiting longer after each failure gives the service room to recover, and spreading the waits apart prevents the clients from arriving in a crowd.

This lesson builds the backoff calculation step by step: the growth itself, the ceiling that keeps it bounded, the randomness that spreads clients apart, and the precedence rule that puts the server's own instructions first. The examples compute every delay exactly, so the behavior can be checked before it reaches a real service.

The Growth Rule

The basic formula doubles a base delay on each attempt. Attempts are numbered from zero, so the first retry waits the base, the second waits twice the base, and so on:

delay(attempt) = base × 2^attempt

The SDK's own values follow this pattern, starting at half a second and capped at eight seconds (Lesson 143). The table below shows the growth with those values, before any cap or randomness:

BASE = 0.5
for attempt in range(7):
    print(f"attempt {attempt}: {BASE * 2 ** attempt:g}s")

Output:

attempt 0: 0.5s
attempt 1: 1s
attempt 2: 2s
attempt 3: 4s
attempt 4: 8s
attempt 5: 16s
attempt 6: 32s

The numbers grow quickly. By the sixth retry, a single wait is half a minute, and a caller that retried indefinitely would eventually wait for hours. A ceiling is therefore necessary.

The Ceiling

A cap sets the largest wait any single attempt can have. Once the doubling would exceed the cap, the wait stays at the cap:

delay(attempt) = min(cap, base × 2^attempt)

With a cap of eight seconds, the same schedule becomes 0.5, 1, 2, 4, 8, 8, 8, and so on. The cap bounds how long a caller waits between attempts, which in turn bounds how long a whole retry sequence can take. The worst-case total follows directly from the schedule, and Lesson 144 showed how to add it to the timeouts.

Choosing the cap is a judgment about the work. A user-facing call might cap at a few seconds, because a longer wait is worse than giving up. A background job might cap at a minute, because patience costs nothing there.

Why Randomness Helps

The schedule above is deterministic, and that is a problem. Suppose a service fails for two seconds, and a thousand clients all start retrying at that moment. They all wait 0.5 seconds, and retry together. They all wait one second more, and retry together again. Each retry wave is as large as the original burst, and the service never gets the quiet it needs.

Randomness breaks the synchronization. If each client waits a slightly different amount, the retries spread across time, and the load on the service becomes smooth. Two common forms exist:

Both forms are written here with a single parameter, jitter, between 0 and 1, and a fraction that stands for the random value. A jitter of 0 turns randomness off, a jitter of 1 is full jitter, and a jitter of 0.5 is equal jitter. Taking the random value as an argument, rather than generating it inside the function, keeps the calculation testable:

"""backoff.py: exponential backoff with optional jitter, computed deterministically for tests."""


def backoff_delay(attempt: int, base: float = 0.5, cap: float = 8.0,
                  jitter: float = 0.0, fraction: float = 0.0) -> float:
    """Delay before retry number `attempt + 1`. `fraction` is a random value in [0, 1)."""
    raw = min(cap, base * 2 ** attempt)
    return raw * (1 - jitter * fraction)


if __name__ == "__main__":
    for attempt in range(7):
        print(f"attempt {attempt}: no jitter {backoff_delay(attempt):g}s, "
              f"equal jitter at fraction 1.0 {backoff_delay(attempt, jitter=0.5, fraction=1.0):g}s")

Output:

attempt 0: no jitter 0.5s, equal jitter at fraction 1.0 0.25s
attempt 1: no jitter 1s, equal jitter at fraction 1.0 0.5s
attempt 2: no jitter 2s, equal jitter at fraction 1.0 1s
attempt 3: no jitter 4s, equal jitter at fraction 1.0 2s
attempt 4: no jitter 8s, equal jitter at fraction 1.0 4s
attempt 5: no jitter 8s, equal jitter at fraction 1.0 4s
attempt 6: no jitter 8s, equal jitter at fraction 1.0 4s

How It Works

1. The cap is applied first, to the raw doubling. Once base × 2^attempt exceeds the cap, the raw value stays at the cap.
2. Jitter scales the capped value down by a random amount. With jitter=0.5 and fraction between 0 and 1, the delay falls between half the raw value and the full raw value. The output shows the lower end, which is what fraction=1.0 produces.
3. The random value is a parameter. Production code passes a value from a random number generator. Tests pass fixed values, and assert the exact result.
4. The schedule stays bounded, because the cap applies before jitter, and jitter only shortens waits. The worst-case wait can never exceed the cap.

Showing the Spread

The reason for jitter is easiest to see by comparison. The simulation below gives a hundred clients the same first retry, once with no jitter and once with equal jitter. The random values come from a small, fixed generator, so the run is deterministic and the results can be checked:

"""jitter_spread.py: how many distinct retry times a hundred clients produce."""

from backoff import backoff_delay


def lcg(seed: int):
    """A small deterministic generator, so the simulation gives the same result every run."""
    state = seed
    while True:
        state = (1103515245 * state + 12345) % 2 ** 31
        yield state / 2 ** 31


clients = 100
random_values = lcg(seed=42)

no_jitter = [backoff_delay(0) for _ in range(clients)]
with_jitter = [backoff_delay(0, jitter=0.5, fraction=next(random_values)) for _ in range(clients)]

print("distinct retry times without jitter:", len(set(no_jitter)))
print("distinct retry times with jitter:   ", len(set(with_jitter)))
print("all within the expected range:", all(0.25 < d <= 0.5 for d in with_jitter))

Output:

distinct retry times without jitter: 1
distinct retry times with jitter:    100
all within the expected range: True

How It Works

1. Without jitter, all hundred clients share one retry time. That is the synchronized burst the pattern exists to prevent.
2. With jitter, every client gets its own time. For the first retry, the formula places every time in the range from just above a quarter second up to half a second, so the spread is a quarter of a second. Later attempts spread over wider ranges, because their raw delays are larger.
3. The range check is exact by construction. A random value in [0, 1) gives a jittered delay of half the raw value at most, and just above a quarter of it at least, so the final check holds for every client, regardless of which values the generator produces.
4. The generator is deterministic. It produces the same sequence every run, so the count of distinct times is the same every time, and the claim can be checked.

Respecting the Server's Instruction

The calculation above is the fallback. When the service sends a retry-after header (Lesson 142), its value is more accurate than any local guess, because the service knows when its window resets. The rule is therefore a precedence: use the header when it is usable, and fall back to the calculated delay only when it is not.

The usable range deserves a check. A header of zero or a negative number is useless, and a very large value may be a mistake, so the function below accepts only values within a sensible range and falls back otherwise:

def next_delay(attempt: int, headers: dict, jitter: float = 0.5, fraction: float = 0.0,
               max_server_wait: float = 60.0) -> float:
    header = headers.get("retry-after")
    if header is not None:
        try:
            seconds = float(header)
        except ValueError:
            seconds = 0.0
        if 0 < seconds <= max_server_wait:
            return seconds
    return backoff_delay(attempt, jitter=jitter, fraction=fraction)


if __name__ == "__main__":
    print(next_delay(2, {"retry-after": "3"}))
    print(next_delay(2, {}))
    print(next_delay(2, {"retry-after": "3600"}))
    print(next_delay(2, {"retry-after": "soon"}, fraction=1.0))

Output:

3.0
2.0
2.0
1.0

How It Works

1. A usable header wins. The server's three-second instruction is used directly, with no jitter added, because the service has already chosen the moment.
2. A missing header falls back to the calculated delay for attempt two, which is two seconds without jitter.
3. A very large header is rejected. An hour-long instruction is outside the sixty-second range, so the calculation takes over. This protects a caller from a bad value that would stall a job for hours.
4. An unreadable header falls back too. The value "soon" cannot be parsed, so it is treated as unusable, and the calculation produces one second with the fraction of one, showing that jitter still applies on the fallback path.

Choosing the Parameters

The base, cap, and jitter form a small set of choices, and each one has a clear effect. A short base suits calls where a quick retry is likely to succeed, such as a brief network blip. A higher cap suits background work, where waiting is cheap. Full jitter spreads retries the most and is a good default for large numbers of clients, while equal jitter keeps a floor under every wait, which is useful when a retry sooner than half a second is pointless.

These parameters are best set from observation. A team that records how long its failures last, and how often a retry at each delay succeeds, can tune the schedule to fit its own service. The defaults in the SDK are a reasonable starting point, not a final answer.

Summary

Exponential backoff doubles the wait after each failure, bounded by a cap, so that a caller backs away from a struggling service instead of pressing on. Randomness spreads clients across time, which turns synchronized bursts into a steady load. The server's retry-after value takes precedence when it is usable, and the calculation covers everything else. The next lesson looks at the other side of reliability: recording what happened, so that failures can be understood after they occur.