Reliability / 3 / 6
Timeouts and retries
A checkout API depends on a rates service that gets slow, then dies. How timeouts, backoff, retry budgets and a circuit breaker keep one sick dependency from taking everything down.
- 13 min
- Intermediate
- 11 steps
- Checkout p95
- 180 ms
- Failed checkouts
- 0.2 %
- Calls to rates
- 400 /s
- Double charges
- 0 /day
- request
- reply
- redelivery
- duplicate
- lost
- call: the reply returns on the same line
Numbers are a model scenario, not measurements
The real request
Checkout API → Rates serviceRequest
GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
Host1: rates.internal
Accept: application/json
Response
HTTP/1.1 200 OK
X-Response-Time2: 85ms
{
"price": 332000,
"currency": "UZS",
"eta_days": 2
}
Host— An internal service, not a public one. It is still a network call that can hang.X-Response-Time— 85 ms today. The code has no idea what to do if it becomes 30 seconds.price— Delivery price in the smallest unit. Checkout cannot finish without some price.
What happens if…?
Put in your own numbers: the result and the formula update at once.
Chance that one call ends in a timeout or an error.
How many retries after the first try.
Each try waits this long, then gives up.
First pause window; it doubles with every retry.
Calls to the service before any retries.
Result
- Mean tries per request
- 1.031
- Load on the service
- 412/s
- Chance of success
- 99.99992%
- Worst-case latency
- 3 900ms
Working
tries = (1 − p^(r+1)) / (1 − p) = (1 − 0.03^4) / (1 − 0.03) = 1.031
load = 400 × 1.031 = 412/s
success = 1 − p^(r+1) = 1 − 0.03^4 = 99.99992%
worst = (r + 1) × T + b × (2^r − 1) = 4 × 800 + 100 × 7 = 3 900 msA simplified teaching model: it shows direction and scale, not a benchmark.
Build it yourselfCheat sheetCompare lessons
Course: 74 lessons
01 Foundations
02 Scaling
03 Data & storage
04 Communication
05 Reliability
06 Architecture
07 Security
08 Shipping
09 Whole systems
10 Business processes
11 Case studies