mufassall

Reliability / 3 / 4

Timeouts and retries

A checkout API depends on a rates service that gets slow, then dies. How timeouts, backoff, retry budgets and a circuit breaker keep one sick dependency from taking everything down.

  • 13 min
  • Intermediate
  • 11 steps
Checkout p95
180 ms
Failed checkouts
0.2 %
Calls to rates
400 /s
Double charges
0 /day
Open in Lab
30%40%
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Checkout API → Rates service

Request

GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
Host1: rates.internal
Accept: application/json

Response

HTTP/1.1 200 OK
X-Response-Time2: 85ms

{
  "price": 332000,
  "currency": "UZS",
  "eta_days": 2
}
  1. Host — An internal service, not a public one. It is still a network call that can hang.
  2. X-Response-Time — 85 ms today. The code has no idea what to do if it becomes 30 seconds.
  3. price — Delivery price in the smallest unit. Checkout cannot finish without some price.
Course: 42 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Storage engines: B-tree and LSM
    4. Database replication
    5. Sharding
    6. Consistent hashing
    7. Unique IDs
    8. CAP: when the link breaks
    9. Consensus and leader election
    10. Transactions
    11. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Batch vs stream
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
  10. 10 Business processes

    1. Payment gateway