mufassall

Cheat sheet · Reliability

Timeouts and retries

13 min · Intermediate · mufassall.dtwin.uz/en/l/timeouts-retries/

← Open the lesson

The problem

A checkout API depends on a rates service that gets slow, then dies. How timeouts, backoff, retry budgets and a circuit breaker keep one sick dependency from taking everything down.

What you learn

  • Why a call without a timeout can freeze every thread
  • Exponential backoff with jitter, and how retries multiply into a storm
  • Retry budgets and the circuit breaker: closed, open, half-open
  • Retry only what is idempotent, or make it so with a key
Final scene
25%40%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_client(["Shop app"])
  n_api["Checkout API"]
  n_rates["Rates service"]
  n_db[("Orders DB")]
  n_pay>"Payment provider"]
  n_client -->|"call"| n_api
  n_api -->|"call"| n_rates
  n_api -->|"call"| n_db
  n_api -->|"call"| n_pay

Mermaid: request sequence

sequenceDiagram
  participant n_api as Checkout API
  participant n_rates as Rates service
  participant n_client as Shop app
  participant n_pay as Payment provider
  Note over n_api,n_rates: Every checkout asks for a delivery price
  n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
  n_rates-->>n_api: HTTP/1.1 200 OK
  Note over n_client,n_api: One slow dependency freezes everything
  n_client->>n_api: POST /v1/checkout HTTP/1.1
  n_api-->>n_client: HTTP/1.1 503 Service Unavailable
  Note over n_api,n_rates: Give up after 800 ms
  n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
  n_rates-->>n_api: timeout: no answer in 800ms
  Note over n_api,n_rates: Instant retries become a storm
  n_api->>n_rates: 10:42:07.112 GET /v1/quote attempt=1 timeout
  Note over n_api,n_rates: Exponential backoff with jitter
  n_api->>n_rates: attempt=1 timeout, sleep 90ms (base 100ms, jitter)
  Note over n_api,n_rates: A retry budget caps the extra load
  n_api->>n_rates: retry budget rates: 40 of 400 per s used (10%)
  Note over n_api,n_rates: Circuit breaker: stop calling a dead service
  n_api->>n_rates: circuit rates: CLOSED -#62; OPEN
  Note over n_api,n_rates: Half-open: one probe decides
  n_api->>n_rates: GET /v1/quote?city=TAS#38;kg=2.4 HTTP/1.1
  n_rates-->>n_api: HTTP/1.1 200 OK in 92ms
  Note over n_api,n_pay: Retrying a charge charges twice
  n_api->>n_pay: POST /v1/charges HTTP/1.1
  n_pay-->>n_api: HTTP/1.1 201 Created
  Note over n_api,n_pay: Retry only idempotent calls
  n_api->>n_pay: POST /v1/charges HTTP/1.1
  n_pay-->>n_api: HTTP/1.1 201 Created

D2

direction: right

n_client: "Shop app" {
  shape: rectangle
}
n_api: "Checkout API" {
  shape: rectangle
}
n_rates: "Rates service" {
  shape: rectangle
}
n_db: "Orders DB" {
  shape: cylinder
}
n_pay: "Payment provider" {
  shape: cloud
}

n_client -> n_api: "call"
n_api -> n_rates: "call"
n_api -> n_db: "call"
n_api -> n_pay: "call"

PlantUML: request sequence

@startuml
participant "Checkout API" as n_api
participant "Rates service" as n_rates
participant "Shop app" as n_client
boundary "Payment provider" as n_pay
note over n_api, n_rates : Every checkout asks for a delivery price
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : HTTP/1.1 200 OK
note over n_client, n_api : One slow dependency freezes everything
n_client -> n_api : POST /v1/checkout HTTP/1.1
n_api --> n_client : HTTP/1.1 503 Service Unavailable
note over n_api, n_rates : Give up after 800 ms
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : timeout: no answer in 800ms
note over n_api, n_rates : Instant retries become a storm
n_api -> n_rates : [retry] 10:42:07.112 GET /v1/quote attempt=1 timeout
note over n_api, n_rates : Exponential backoff with jitter
n_api -> n_rates : [retry] attempt=1 timeout, sleep 90ms (base 100ms, jitter)
note over n_api, n_rates : A retry budget caps the extra load
n_api -> n_rates : [retry] retry budget rates: 40 of 400 per s used (10%)
note over n_api, n_rates : Circuit breaker: stop calling a dead service
n_api -> n_rates : circuit rates: CLOSED -> OPEN
note over n_api, n_rates : Half-open: one probe decides
n_api -> n_rates : GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
n_rates --> n_api : HTTP/1.1 200 OK in 92ms
note over n_api, n_pay : Retrying a charge charges twice
n_api -> n_pay : [duplicate] POST /v1/charges HTTP/1.1
n_pay --> n_api : HTTP/1.1 201 Created
note over n_api, n_pay : Retry only idempotent calls
n_api -> n_pay : POST /v1/charges HTTP/1.1
n_pay --> n_api : HTTP/1.1 201 Created
@enduml

PlantUML: components

@startuml
left to right direction
agent "Shop app" as n_client
component "Checkout API" as n_api
component "Rates service" as n_rates
database "Orders DB" as n_db
cloud "Payment provider" as n_pay
n_client --> n_api : call
n_api --> n_rates : call
n_api --> n_db : call
n_api --> n_pay : call
@enduml

Recap

Five rules for calling something that can fail.

1) Every network call has a timeout set from real latency. 2) Retry in one layer, with exponential backoff and jitter. 3) A retry budget caps the extra load. 4) A circuit breaker stops calls to a dead service and probes it in half-open. 5) Retry only idempotent calls; give the rest an idempotency key.

Numbers: at the start and at the end

MetricStartEnd
Checkout p95180 ms190 ms
Failed checkouts0.2%0.2%
Calls to rates400/s400/s
Double charges0/day0/day

Steps

  1. Case Every checkout asks for a delivery price

    The shop app sends a checkout to the API.

  2. Problem One slow dependency freezes everything

    The rates service starts answering in 30 seconds instead of 85 ms.

  3. Fix Give up after 800 ms

    The API now waits at most 800 ms for rates: a short connect timeout plus a read timeout.

  4. Problem Instant retries become a storm

    To cut the fallbacks, someone added three instant retries in the API.

  5. Fix Exponential backoff with jitter

    Only the API retries now; the app does not.

  6. Fix A retry budget caps the extra load

    Backoff spaces retries out, but in a long outage they still add up.

  7. Fix Circuit breaker: stop calling a dead service

    Rates is fully down.

  8. Case Half-open: one probe decides

    After 30 seconds the breaker goes half-open and lets a single real call through.

  9. Problem Retrying a charge charges twice

    The same retry code is reused for the payment call.

  10. Fix Retry only idempotent calls

    GET for a price is safe to repeat.

  11. Recap Recap

    Five rules for calling something that can fail.

Terms

Timeout
How long to wait for an answer before giving up. Without one, a single slow dependency can tie up every thread and freeze everything; set it from real latency.
Exponential backoff
The wait before each retry doubles. Instant retries no longer grow into a storm that buries a struggling service.
Jitter
A random part added to a wait so that many clients do not retry at the same instant. It is also added to cache TTLs so keys do not expire together.
Circuit breaker
When the failure rate passes a threshold, the breaker opens and for a while does not call the dead service at all, returning a fallback. Then, half-open, one probe call decides whether it closes or opens again.