Sweaty Imposter

Deadline / Retry Budget

TechniqueReliabilitySeniorStaffUpdated
Bound the total time spent retrying, not the number of attempts. Each retry costs from the same budget.

When to Use Deadline / Retry Budget

Any call inside a larger request that itself has a timeout. Counting attempts is the wrong unit: 5 retries with backoff can blow past your caller's 200ms service level agreement (SLA).

In Practice

gRPC's context deadlines. Each downstream remote procedure call (RPC) inherits a deadline; retries must finish before it expires. Envoy and Istio implement the same with a x-envoy-upstream-rq-timeout-ms header.

How Deadline / Retry Budget Comes Up in Interviews

Scenario

Your gRPC handler has a 200ms service level agreement (SLA) to its caller, but it sometimes blows past 2 seconds because a downstream call retries 5 times with backoff. Counting retries is clearly the wrong unit. What is?

A deadline (retry budget) bounds total time spent on the operation; every retry consumes from the same shared budget so the caller's service level agreement (SLA) is respected. Backoff alone says nothing about total elapsed time.

Deadlines (called retry budgets in some traditions) make 'total time' the unit of cost instead of 'number of attempts.' gRPC propagates the deadline through the call chain via the grpc-timeout header so every downstream service knows the absolute clock-time it has left, and any retry that would blow past it is suppressed. The classic anti-pattern is 'retry 3 times with exponential backoff' inside a 200ms SLA: even with a 50ms base, three retries can take 350ms. Envoy, Istio, and Linkerd all surface deadline-aware retry policies via x-envoy-upstream-rq-timeout-ms and equivalent headers. Without deadlines, retries silently turn from 'reliability feature' into 'SLA-violating amplifier.'