Cheat sheet · Reliability
Observability
The problem
Checkout takes 2.4 seconds and every service says it is fine. How logs, a propagated request id, RED and USE metrics, an alert on p95 and a distributed trace find the slow part.
What you learn
- What logs, metrics and traces each answer, and what they cannot
- Why one request or trace id must travel through every service
- RED metrics for services, USE metrics for resources, and alerting on p95
- Reading a trace to find the slow span, and sampling to keep the cost down
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_client(["Customer app"])
n_api["API"]
n_orders["Orders service"]
n_stock["Inventory service"]
n_db[("Orders DB")]
n_logs[("Log store")]
n_tsdb[("Metrics DB")]
n_oncall(("On-call engineer"))
n_tracer[("Trace store")]
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb -.-> n_oncall
n_api --> n_tracer
n_orders --> n_tracerMermaid: request sequence
sequenceDiagram
participant n_client as Customer app
participant n_api as API
participant n_orders as Orders service
participant n_logs as Log store
participant n_tsdb as Metrics DB
participant n_db as Orders DB
actor n_oncall as On-call engineer
participant n_tracer as Trace store
participant n_stock as Inventory service
Note over n_client,n_api: Slow, but nothing is broken
n_client->>n_api: POST /v1/orders HTTP/1.1
n_api-->>n_client: HTTP/1.1 201 Created
Note over n_orders,n_logs: Structured logs in one place
n_orders->>n_logs: orders stdout → log shipper
Note over n_api,n_orders: One id for the whole request
n_api->>n_orders: POST /internal/orders HTTP/1.1
n_orders-->>n_api: log line in orders
Note over n_orders,n_tsdb: RED metrics: rate, errors, duration
n_orders->>n_tsdb: GET /metrics HTTP/1.1
n_tsdb-->>n_orders: HTTP/1.1 200 OK
Note over n_db,n_tsdb: USE metrics: is the database the problem?
n_db->>n_tsdb: GET /metrics HTTP/1.1
n_tsdb-->>n_db: HTTP/1.1 200 OK
Note over n_tsdb,n_oncall: Alert on p95, not on the average
n_tsdb->>n_oncall: ALERT OrdersSlowP95 firing
Note over n_api,n_orders: Traces: time spent in every hop
n_api->>n_orders: POST /internal/orders HTTP/1.1
Note over n_orders,n_tracer: Finding the slow span
n_orders->>n_tracer: trace 4bf92f35 POST /v1/orders 2310ms
Note over n_orders,n_stock: One call for the whole cart
n_orders->>n_stock: GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock-->>n_orders: HTTP/1.1 200 OK
Note over n_api,n_tracer: Traces are expensive: sample
n_api->>n_tracer: sampler: keep 10% of traces, all errors, all over 1sD2
direction: right
n_client: "Customer app" {
shape: rectangle
}
n_api: "API" {
shape: rectangle
}
n_orders: "Orders service" {
shape: rectangle
}
n_stock: "Inventory service" {
shape: rectangle
}
n_db: "Orders DB" {
shape: cylinder
}
n_logs: "Log store" {
shape: stored_data
}
n_tsdb: "Metrics DB" {
shape: cylinder
}
n_oncall: "On-call engineer" {
shape: person
}
n_tracer: "Trace store" {
shape: stored_data
}
n_client -> n_api
n_api -> n_orders
n_orders -> n_stock
n_orders -> n_db
n_api -> n_logs
n_orders -> n_logs
n_api -> n_tsdb
n_orders -> n_tsdb
n_db -> n_tsdb
n_tsdb -> n_oncall {
style.stroke-dash: 3
}
n_api -> n_tracer
n_orders -> n_tracerPlantUML: request sequence
@startuml
participant "Customer app" as n_client
participant "API" as n_api
participant "Orders service" as n_orders
database "Log store" as n_logs
database "Metrics DB" as n_tsdb
database "Orders DB" as n_db
actor "On-call engineer" as n_oncall
database "Trace store" as n_tracer
participant "Inventory service" as n_stock
note over n_client, n_api : Slow, but nothing is broken
n_client -> n_api : POST /v1/orders HTTP/1.1
n_api --> n_client : HTTP/1.1 201 Created
note over n_orders, n_logs : Structured logs in one place
n_orders -> n_logs : orders stdout → log shipper
note over n_api, n_orders : One id for the whole request
n_api -> n_orders : POST /internal/orders HTTP/1.1
n_orders --> n_api : log line in orders
note over n_orders, n_tsdb : RED metrics: rate, errors, duration
n_orders -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_orders : HTTP/1.1 200 OK
note over n_db, n_tsdb : USE metrics: is the database the problem?
n_db -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_db : HTTP/1.1 200 OK
note over n_tsdb, n_oncall : Alert on p95, not on the average
n_tsdb -> n_oncall : ALERT OrdersSlowP95 firing
note over n_api, n_orders : Traces: time spent in every hop
n_api -> n_orders : POST /internal/orders HTTP/1.1
note over n_orders, n_tracer : Finding the slow span
n_orders -> n_tracer : trace 4bf92f35 POST /v1/orders 2310ms
note over n_orders, n_stock : One call for the whole cart
n_orders -> n_stock : GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock --> n_orders : HTTP/1.1 200 OK
note over n_api, n_tracer : Traces are expensive: sample
n_api -> n_tracer : sampler: keep 10% of traces, all errors, all over 1s
@endumlPlantUML: components
@startuml
left to right direction
agent "Customer app" as n_client
component "API" as n_api
component "Orders service" as n_orders
component "Inventory service" as n_stock
database "Orders DB" as n_db
storage "Log store" as n_logs
database "Metrics DB" as n_tsdb
actor "On-call engineer" as n_oncall
storage "Trace store" as n_tracer
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb ..> n_oncall
n_api --> n_tracer
n_orders --> n_tracer
@endumlRecap
Logs say what happened, metrics say how much and how often, traces say where the time went.
1) Log structured JSON to one place. 2) Create one request or trace id at the edge and forward it everywhere. 3) RED for services, USE for resources. 4) Alert on p95 and errors over minutes, not on averages. 5) Use traces to find the slow span, and sample them to control cost.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Checkout p95 | 2 400 ms | 420 ms |
| Time to find cause | 5 400 s | 120 s |
| Telemetry per day | 0.4 GB | 2.5 GB |
Steps
- Problem Slow, but nothing is broken
Customers say checkout hangs.
- Fix Structured logs in one place
Every service now writes one JSON object per event to stdout.
- Fix One id for the whole request
The API creates a request id when a request arrives and passes it on in a header.
- Fix RED metrics: rate, errors, duration
Logs describe single events.
- Case USE metrics: is the database the problem?
For resources, such as a database, a disk or a pool, look at USE: Utilisation, Saturation, Errors.
- Fix Alert on p95, not on the average
Until now, customers noticed first.
- Fix Traces: time spent in every hop
The request id is upgraded to a trace.
- Case Finding the slow span
Open one slow trace.
- Fix One call for the whole cart
Orders now asks inventory once for all items in the cart.
- Case Traces are expensive: sample
Tracing every request produces 12 GB per day, most of it fast, boring requests.
- Recap Recap
Logs say what happened, metrics say how much and how often, traces say where the time went.
Terms
- Observability (logs, metrics, traces)
- Logs say what happened, metrics say how much and how often, traces say where the time went. Use RED metrics for services, USE for resources, and alert on p95, not on averages.
- Trace and span
- Each service records a span for a request: what it did, when it started and how long it took. Spans share a trace id and are assembled into one tree, so the slow hop stands out.