mufassall

Reliability / 4 / 4

Observability

Checkout takes 2.4 seconds and every service says it is fine. How logs, a propagated request id, RED and USE metrics, an alert on p95 and a distributed trace find the slow part.

  • 13 min
  • Intermediate
  • 11 steps
Checkout p95
2 400 ms
Time to find cause
5 400 s
Telemetry per day
0.4 GB
Open in Lab
70%55%
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Customer app → API

Request

POST /v1/orders HTTP/1.1
Host: api.example.com

Response

HTTP/1.1 201 Created1
X-Response-Time2: 2380ms
  1. 201 Created — Success. Nothing failed, so no error log and no exception to follow.
  2. X-Response-Time — 2.4 seconds. The only clue, and it says nothing about where the time went.
Course: 42 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Storage engines: B-tree and LSM
    4. Database replication
    5. Sharding
    6. Consistent hashing
    7. Unique IDs
    8. CAP: when the link breaks
    9. Consensus and leader election
    10. Transactions
    11. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Batch vs stream
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
  10. 10 Business processes

    1. Payment gateway