Cheat sheet · Reliability
Timeouts and retries
The problem
A checkout API depends on a rates service that gets slow, then dies. How timeouts, backoff, retry budgets and a circuit breaker keep one sick dependency from taking everything down.
What you learn
- Why a call without a timeout can freeze every thread
- Exponential backoff with jitter, and how retries multiply into a storm
- Retry budgets and the circuit breaker: closed, open, half-open
- Retry only what is idempotent, or make it so with a key
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_client(["Shop app"])
n_api["Checkout API"]
n_rates["Rates service"]
n_db[("Orders DB")]
n_pay>"Payment provider"]
n_client -->|"call"| n_api
n_api -->|"call"| n_rates
n_api -->|"call"| n_db
n_api -->|"call"| n_payMermaid: request sequence
sequenceDiagram
participant n_api as Checkout API
participant n_rates as Rates service
participant n_client as Shop app
participant n_pay as Payment provider
Note over n_api,n_rates: Every checkout asks for a delivery price
n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
n_rates-->>n_api: HTTP/1.1 200 OK
Note over n_client,n_api: One slow dependency freezes everything
n_client->>n_api: POST /v1/checkout HTTP/1.1
n_api-->>n_client: HTTP/1.1 503 Service Unavailable
Note over n_api,n_rates: Give up after 800 ms
n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
n_rates-->>n_api: timeout: no answer in 800ms
Note over n_api,n_rates: Instant retries become a storm
n_api->>n_rates: 10:42:07.112 GET /v1/quote attempt=1 timeout
Note over n_api,n_rates: Exponential backoff with jitter
n_api->>n_rates: attempt=1 timeout, sleep 90ms (base 100ms, jitter)
Note over n_api,n_rates: A retry budget caps the extra load
n_api->>n_rates: retry budget rates: 40 of 400 per s used (10%)
Note over n_api,n_rates: Circuit breaker: stop calling a dead service
n_api->>n_rates: circuit rates: CLOSED -#62; OPEN
Note over n_api,n_rates: Half-open: one probe decides
n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
n_rates-->>n_api: HTTP/1.1 200 OK in 92ms
Note over n_api,n_pay: Retrying a charge charges twice
n_api->>n_pay: POST /v1/charges HTTP/1.1
n_pay-->>n_api: HTTP/1.1 201 Created
Note over n_api,n_pay: Retry only idempotent calls
n_api->>n_pay: POST /v1/charges HTTP/1.1
n_pay-->>n_api: HTTP/1.1 201 CreatedD2
direction: right
n_client: "Shop app" {
shape: rectangle
}
n_api: "Checkout API" {
shape: rectangle
}
n_rates: "Rates service" {
shape: rectangle
}
n_db: "Orders DB" {
shape: cylinder
}
n_pay: "Payment provider" {
shape: cloud
}
n_client -> n_api: "call"
n_api -> n_rates: "call"
n_api -> n_db: "call"
n_api -> n_pay: "call"PlantUML: request sequence
@startuml
participant "Checkout API" as n_api
participant "Rates service" as n_rates
participant "Shop app" as n_client
boundary "Payment provider" as n_pay
note over n_api, n_rates : Every checkout asks for a delivery price
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : HTTP/1.1 200 OK
note over n_client, n_api : One slow dependency freezes everything
n_client -> n_api : POST /v1/checkout HTTP/1.1
n_api --> n_client : HTTP/1.1 503 Service Unavailable
note over n_api, n_rates : Give up after 800 ms
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : timeout: no answer in 800ms
note over n_api, n_rates : Instant retries become a storm
n_api -> n_rates : [retry] 10:42:07.112 GET /v1/quote attempt=1 timeout
note over n_api, n_rates : Exponential backoff with jitter
n_api -> n_rates : [retry] attempt=1 timeout, sleep 90ms (base 100ms, jitter)
note over n_api, n_rates : A retry budget caps the extra load
n_api -> n_rates : [retry] retry budget rates: 40 of 400 per s used (10%)
note over n_api, n_rates : Circuit breaker: stop calling a dead service
n_api -> n_rates : circuit rates: CLOSED -> OPEN
note over n_api, n_rates : Half-open: one probe decides
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : HTTP/1.1 200 OK in 92ms
note over n_api, n_pay : Retrying a charge charges twice
n_api -> n_pay : [duplicate] POST /v1/charges HTTP/1.1
n_pay --> n_api : HTTP/1.1 201 Created
note over n_api, n_pay : Retry only idempotent calls
n_api -> n_pay : POST /v1/charges HTTP/1.1
n_pay --> n_api : HTTP/1.1 201 Created
@endumlPlantUML: components
@startuml
left to right direction
agent "Shop app" as n_client
component "Checkout API" as n_api
component "Rates service" as n_rates
database "Orders DB" as n_db
cloud "Payment provider" as n_pay
n_client --> n_api : call
n_api --> n_rates : call
n_api --> n_db : call
n_api --> n_pay : call
@endumlRecap
Five rules for calling something that can fail.
1) Every network call has a timeout set from real latency. 2) Retry in one layer, with exponential backoff and jitter. 3) A retry budget caps the extra load. 4) A circuit breaker stops calls to a dead service and probes it in half-open. 5) Retry only idempotent calls; give the rest an idempotency key.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Checkout p95 | 180 ms | 190 ms |
| Failed checkouts | 0.2% | 0.2% |
| Calls to rates | 400/s | 400/s |
| Double charges | 0/day | 0/day |
Steps
- Case Every checkout asks for a delivery price
The shop app sends a checkout to the API.
- Problem One slow dependency freezes everything
The rates service starts answering in 30 seconds instead of 85 ms.
- Fix Give up after 800 ms
The API now waits at most 800 ms for rates: a short connect timeout plus a read timeout.
- Problem Instant retries become a storm
To cut the fallbacks, someone added three instant retries in the API.
- Fix Exponential backoff with jitter
Only the API retries now; the app does not.
- Fix A retry budget caps the extra load
Backoff spaces retries out, but in a long outage they still add up.
- Fix Circuit breaker: stop calling a dead service
Rates is fully down.
- Case Half-open: one probe decides
After 30 seconds the breaker goes half-open and lets a single real call through.
- Problem Retrying a charge charges twice
The same retry code is reused for the payment call.
- Fix Retry only idempotent calls
GET for a price is safe to repeat.
- Recap Recap
Five rules for calling something that can fail.
Terms
- Timeout
- How long to wait for an answer before giving up. Without one, a single slow dependency can tie up every thread and freeze everything; set it from real latency.
- Exponential backoff
- The wait before each retry doubles. Instant retries no longer grow into a storm that buries a struggling service.
- Jitter
- A random part added to a wait so that many clients do not retry at the same instant. It is also added to cache TTLs so keys do not expire together.
- Circuit breaker
- When the failure rate passes a threshold, the breaker opens and for a while does not call the dead service at all, returning a fallback. Then, half-open, one probe call decides whether it closes or opens again.