mufassall

Cheat sheet · Reliability

Observability

13 min · Intermediate · mufassall.dtwin.uz/en/l/observability/

← Open the lesson

The problem

Checkout takes 2.4 seconds and every service says it is fine. How logs, a propagated request id, RED and USE metrics, an alert on p95 and a distributed trace find the slow part.

What you learn

  • What logs, metrics and traces each answer, and what they cannot
  • Why one request or trace id must travel through every service
  • RED metrics for services, USE metrics for resources, and alerting on p95
  • Reading a trace to find the slow span, and sampling to keep the cost down
Final scene
35%30%41%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_client(["Customer app"])
  n_api["API"]
  n_orders["Orders service"]
  n_stock["Inventory service"]
  n_db[("Orders DB")]
  n_logs[("Log store")]
  n_tsdb[("Metrics DB")]
  n_oncall(("On-call engineer"))
  n_tracer[("Trace store")]
  n_client --> n_api
  n_api --> n_orders
  n_orders --> n_stock
  n_orders --> n_db
  n_api --> n_logs
  n_orders --> n_logs
  n_api --> n_tsdb
  n_orders --> n_tsdb
  n_db --> n_tsdb
  n_tsdb -.-> n_oncall
  n_api --> n_tracer
  n_orders --> n_tracer

Mermaid: request sequence

sequenceDiagram
  participant n_client as Customer app
  participant n_api as API
  participant n_orders as Orders service
  participant n_logs as Log store
  participant n_tsdb as Metrics DB
  participant n_db as Orders DB
  actor n_oncall as On-call engineer
  participant n_tracer as Trace store
  participant n_stock as Inventory service
  Note over n_client,n_api: Slow, but nothing is broken
  n_client->>n_api: POST /v1/orders HTTP/1.1
  n_api-->>n_client: HTTP/1.1 201 Created
  Note over n_orders,n_logs: Structured logs in one place
  n_orders->>n_logs: orders stdout → log shipper
  Note over n_api,n_orders: One id for the whole request
  n_api->>n_orders: POST /internal/orders HTTP/1.1
  n_orders-->>n_api: log line in orders
  Note over n_orders,n_tsdb: RED metrics: rate, errors, duration
  n_orders->>n_tsdb: GET /metrics HTTP/1.1
  n_tsdb-->>n_orders: HTTP/1.1 200 OK
  Note over n_db,n_tsdb: USE metrics: is the database the problem?
  n_db->>n_tsdb: GET /metrics HTTP/1.1
  n_tsdb-->>n_db: HTTP/1.1 200 OK
  Note over n_tsdb,n_oncall: Alert on p95, not on the average
  n_tsdb->>n_oncall: ALERT OrdersSlowP95 firing
  Note over n_api,n_orders: Traces: time spent in every hop
  n_api->>n_orders: POST /internal/orders HTTP/1.1
  Note over n_orders,n_tracer: Finding the slow span
  n_orders->>n_tracer: trace 4bf92f35 POST /v1/orders 2310ms
  Note over n_orders,n_stock: One call for the whole cart
  n_orders->>n_stock: GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
  n_stock-->>n_orders: HTTP/1.1 200 OK
  Note over n_api,n_tracer: Traces are expensive: sample
  n_api->>n_tracer: sampler: keep 10% of traces, all errors, all over 1s

D2

direction: right

n_client: "Customer app" {
  shape: rectangle
}
n_api: "API" {
  shape: rectangle
}
n_orders: "Orders service" {
  shape: rectangle
}
n_stock: "Inventory service" {
  shape: rectangle
}
n_db: "Orders DB" {
  shape: cylinder
}
n_logs: "Log store" {
  shape: stored_data
}
n_tsdb: "Metrics DB" {
  shape: cylinder
}
n_oncall: "On-call engineer" {
  shape: person
}
n_tracer: "Trace store" {
  shape: stored_data
}

n_client -> n_api
n_api -> n_orders
n_orders -> n_stock
n_orders -> n_db
n_api -> n_logs
n_orders -> n_logs
n_api -> n_tsdb
n_orders -> n_tsdb
n_db -> n_tsdb
n_tsdb -> n_oncall {
  style.stroke-dash: 3
}
n_api -> n_tracer
n_orders -> n_tracer

PlantUML: request sequence

@startuml
participant "Customer app" as n_client
participant "API" as n_api
participant "Orders service" as n_orders
database "Log store" as n_logs
database "Metrics DB" as n_tsdb
database "Orders DB" as n_db
actor "On-call engineer" as n_oncall
database "Trace store" as n_tracer
participant "Inventory service" as n_stock
note over n_client, n_api : Slow, but nothing is broken
n_client -> n_api : POST /v1/orders HTTP/1.1
n_api --> n_client : HTTP/1.1 201 Created
note over n_orders, n_logs : Structured logs in one place
n_orders -> n_logs : orders stdout → log shipper
note over n_api, n_orders : One id for the whole request
n_api -> n_orders : POST /internal/orders HTTP/1.1
n_orders --> n_api : log line in orders
note over n_orders, n_tsdb : RED metrics: rate, errors, duration
n_orders -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_orders : HTTP/1.1 200 OK
note over n_db, n_tsdb : USE metrics: is the database the problem?
n_db -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_db : HTTP/1.1 200 OK
note over n_tsdb, n_oncall : Alert on p95, not on the average
n_tsdb -> n_oncall : ALERT OrdersSlowP95 firing
note over n_api, n_orders : Traces: time spent in every hop
n_api -> n_orders : POST /internal/orders HTTP/1.1
note over n_orders, n_tracer : Finding the slow span
n_orders -> n_tracer : trace 4bf92f35 POST /v1/orders 2310ms
note over n_orders, n_stock : One call for the whole cart
n_orders -> n_stock : GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock --> n_orders : HTTP/1.1 200 OK
note over n_api, n_tracer : Traces are expensive: sample
n_api -> n_tracer : sampler: keep 10% of traces, all errors, all over 1s
@enduml

PlantUML: components

@startuml
left to right direction
agent "Customer app" as n_client
component "API" as n_api
component "Orders service" as n_orders
component "Inventory service" as n_stock
database "Orders DB" as n_db
storage "Log store" as n_logs
database "Metrics DB" as n_tsdb
actor "On-call engineer" as n_oncall
storage "Trace store" as n_tracer
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb ..> n_oncall
n_api --> n_tracer
n_orders --> n_tracer
@enduml

Recap

Logs say what happened, metrics say how much and how often, traces say where the time went.

1) Log structured JSON to one place. 2) Create one request or trace id at the edge and forward it everywhere. 3) RED for services, USE for resources. 4) Alert on p95 and errors over minutes, not on averages. 5) Use traces to find the slow span, and sample them to control cost.

Numbers: at the start and at the end

MetricStartEnd
Checkout p952 400 ms420 ms
Time to find cause5 400 s120 s
Telemetry per day0.4 GB2.5 GB

Steps

  1. Problem Slow, but nothing is broken

    Customers say checkout hangs.

  2. Fix Structured logs in one place

    Every service now writes one JSON object per event to stdout.

  3. Fix One id for the whole request

    The API creates a request id when a request arrives and passes it on in a header.

  4. Fix RED metrics: rate, errors, duration

    Logs describe single events.

  5. Case USE metrics: is the database the problem?

    For resources, such as a database, a disk or a pool, look at USE: Utilisation, Saturation, Errors.

  6. Fix Alert on p95, not on the average

    Until now, customers noticed first.

  7. Fix Traces: time spent in every hop

    The request id is upgraded to a trace.

  8. Case Finding the slow span

    Open one slow trace.

  9. Fix One call for the whole cart

    Orders now asks inventory once for all items in the cart.

  10. Case Traces are expensive: sample

    Tracing every request produces 12 GB per day, most of it fast, boring requests.

  11. Recap Recap

    Logs say what happened, metrics say how much and how often, traces say where the time went.

Terms

Observability (logs, metrics, traces)
Logs say what happened, metrics say how much and how often, traces say where the time went. Use RED metrics for services, USE for resources, and alert on p95, not on averages.
Trace and span
Each service records a span for a request: what it did, when it started and how long it took. Spans share a trace id and are assembled into one tree, so the slow hop stands out.