mufassall

Reliability / 3 / 6

Timeouts and retries

A checkout API depends on a rates service that gets slow, then dies. How timeouts, backoff, retry budgets and a circuit breaker keep one sick dependency from taking everything down.

  • 13 min
  • Intermediate
  • 11 steps
Checkout p95
180 ms
Failed checkouts
0.2 %
Calls to rates
400 /s
Double charges
0 /day
Open in Lab

30%40%
  • request
  • reply
  • redelivery
  • duplicate
  • lost
  • call: the reply returns on the same line

Numbers are a model scenario, not measurements

The real request

Checkout API → Rates service

Request

GET /v1/quote?city=TAS&kg=2.4 HTTP/1.1
Host1: rates.internal
Accept: application/json

Response

HTTP/1.1 200 OK
X-Response-Time2: 85ms

{
  "price": 332000,
  "currency": "UZS",
  "eta_days": 2
}
  1. Host — An internal service, not a public one. It is still a network call that can hang.
  2. X-Response-Time — 85 ms today. The code has no idea what to do if it becomes 30 seconds.
  3. price — Delivery price in the smallest unit. Checkout cannot finish without some price.

What happens if…?

Put in your own numbers: the result and the formula update at once.

Chance that one call ends in a timeout or an error.

How many retries after the first try.

Each try waits this long, then gives up.

First pause window; it doubles with every retry.

Calls to the service before any retries.

Result

Mean tries per request
1.031
Load on the service
412/s
Chance of success
99.99992%
Worst-case latency
3 900ms

Working

tries = (1 − p^(r+1)) / (1 − p) = (1 − 0.03^4) / (1 − 0.03) = 1.031
load = 400 × 1.031 = 412/s
success = 1 − p^(r+1) = 1 − 0.03^4 = 99.99992%
worst = (r + 1) × T + b × (2^r − 1) = 4 × 800 + 100 × 7 = 3 900 ms

A simplified teaching model: it shows direction and scale, not a benchmark.

Build it yourselfCheat sheetCompare lessons

Course: 74 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. Cache invalidation
    6. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Geo search: who is nearby
    4. Full-text search
    5. Vector search
    6. Storage engines: B-tree and LSM
    7. Bloom filters
    8. Database replication
    9. Sharding
    10. Consistent hashing
    11. Unique IDs
    12. CAP: when the link breaks
    13. Consensus and leader election
    14. Transactions
    15. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
    6. Sending webhooks reliably
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
    5. SLOs and error budgets
    6. Backpressure
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Change data capture (CDC)
    5. Transactional outbox
    6. Distributed locks
    7. Batch vs stream
    8. Multi-tenancy
    9. Event sourcing
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
    4. Secrets management
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
    3. Zero-downtime migrations
    4. Feature flags
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
    5. Design a video streaming site
    6. Design search autocomplete
    7. Design a job scheduler
  10. 10 Business processes

    1. Payment gateway
    2. Money and rounding
    3. Double-entry ledger
    4. Payment reconciliation
    5. Subscription billing
    6. Electronic digital signature
    7. Document lifecycle
    8. Audit log
    9. Notification delivery
    10. Bulk import
    11. Refunds and chargebacks
    12. Gapless document numbering
  11. 11 Case studies

    1. Case: how Discord stores messages
    2. Case: Figma multiplayer editing