Cheat sheet · Reliability
SLOs and error budgets
The problem
A taxi app's booking API serves about 100 requests a second. The health check says 100% while riders see errors. How to measure reliability the way users feel it, agree on a target, turn the gap into an error budget, and page people only when that budget is really burning.
What you learn
- SLI, SLO and SLA: what is measured, what is the target, what is promised
- Error budget math: 99.9% over 30 days is about 43 minutes or 259 200 failed requests
- Burn rate and multiwindow alerts that page fast and stay quiet on blips
- Release freezes when the budget is spent, and latency SLOs by percentile
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_riders(["Riders"])
n_lb{"Load balancer"}
n_api["Booking API"]
n_db[("Trips DB")]
n_monitor["Monitoring"]
n_pipeline("Release pipeline")
n_pager(("On-call"))
n_riders --> n_lb
n_lb --> n_api
n_api --> n_db
n_lb --> n_monitor
n_pipeline --> n_api
n_monitor --> n_pager
n_pipeline --> n_monitorMermaid: request sequence
sequenceDiagram
participant n_monitor as Monitoring
participant n_api as Booking API
participant n_lb as Load balancer
participant n_riders as Riders
participant n_pipeline as Release pipeline
actor n_pager as On-call
Note over n_monitor,n_api: The health check is green, riders see errors
n_monitor->>n_api: GET /healthz HTTP/1.1
n_api-->>n_monitor: HTTP/1.1 200 OK
Note over n_lb,n_monitor: SLI: count good requests where users meet you
n_lb->>n_monitor: lb access log, POST /rides, 10:00-10:01
Note over n_lb,n_monitor: SLO: a target, and the budget it leaves
n_lb->>n_monitor: slo ride-booking
Note over n_riders,n_lb: SLA: the promise with money attached
n_riders->>n_lb: SLA, corporate rides contract
Note over n_pipeline,n_api: A release burns the budget fifty times too fast
n_pipeline->>n_api: deploy ride-api v412 to 100%
n_api-->>n_pipeline: error rate 5.0%, the SLO allows 0.1%
Note over n_monitor,n_pager: Two naive alerts: too noisy and too late
n_monitor->>n_pager: rule A: error_rate_5m #62; 0.1% -#62; page
Note over n_monitor,n_pager: Burn-rate alerts with two windows
n_monitor->>n_pager: ALERT BudgetBurnFast (page)
n_pager-->>n_monitor: v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
Note over n_pipeline,n_monitor: Budget spent: releases stop
n_pipeline->>n_monitor: GET /slo/ride-booking/budget HTTP/1.1
n_monitor-->>n_pipeline: HTTP/1.1 200 OK
Note over n_lb,n_monitor: A latency SLO is a percentile, not an average
n_lb->>n_monitor: histogram ride_request_duration, last 30 daysD2
direction: right
n_riders: "Riders" {
shape: rectangle
}
n_lb: "Load balancer" {
shape: diamond
}
n_api: "Booking API" {
shape: rectangle
}
n_db: "Trips DB" {
shape: cylinder
}
n_monitor: "Monitoring" {
shape: rectangle
}
n_pipeline: "Release pipeline" {
shape: rectangle
style.multiple: true
}
n_pager: "On-call" {
shape: person
}
n_riders -> n_lb
n_lb -> n_api
n_api -> n_db
n_lb -> n_monitor
n_pipeline -> n_api
n_monitor -> n_pager
n_pipeline -> n_monitorPlantUML: request sequence
@startuml
participant "Monitoring" as n_monitor
participant "Booking API" as n_api
participant "Load balancer" as n_lb
participant "Riders" as n_riders
participant "Release pipeline" as n_pipeline
actor "On-call" as n_pager
note over n_monitor, n_api : The health check is green, riders see errors
n_monitor -> n_api : GET /healthz HTTP/1.1
n_api --> n_monitor : HTTP/1.1 200 OK
note over n_lb, n_monitor : SLI: count good requests where users meet you
n_lb -> n_monitor : lb access log, POST /rides, 10:00-10:01
note over n_lb, n_monitor : SLO: a target, and the budget it leaves
n_lb -> n_monitor : slo ride-booking
note over n_riders, n_lb : SLA: the promise with money attached
n_riders -> n_lb : SLA, corporate rides contract
note over n_pipeline, n_api : A release burns the budget fifty times too fast
n_pipeline -> n_api : deploy ride-api v412 to 100%
n_api --> n_pipeline : error rate 5.0%, the SLO allows 0.1%
note over n_monitor, n_pager : Two naive alerts: too noisy and too late
n_monitor -> n_pager : rule A: error_rate_5m > 0.1% -> page
note over n_monitor, n_pager : Burn-rate alerts with two windows
n_monitor -> n_pager : ALERT BudgetBurnFast (page)
n_pager --> n_monitor : v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
note over n_pipeline, n_monitor : Budget spent: releases stop
n_pipeline -> n_monitor : GET /slo/ride-booking/budget HTTP/1.1
n_monitor --> n_pipeline : HTTP/1.1 200 OK
note over n_lb, n_monitor : A latency SLO is a percentile, not an average
n_lb -> n_monitor : histogram ride_request_duration, last 30 days
@endumlPlantUML: components
@startuml
left to right direction
agent "Riders" as n_riders
node "Load balancer" as n_lb
component "Booking API" as n_api
database "Trips DB" as n_db
component "Monitoring" as n_monitor
collections "Release pipeline" as n_pipeline
actor "On-call" as n_pager
n_riders --> n_lb
n_lb --> n_api
n_api --> n_db
n_lb --> n_monitor
n_pipeline --> n_api
n_monitor --> n_pager
n_pipeline --> n_monitor
@endumlRecap
Six rules for SLOs.
1) Measure the SLI where users meet you: good requests over valid ones, at the load balancer. 2) Pick an SLO below 100%; the rest is the error budget (99.9% over 30 days is about 43 minutes). 3) Keep the SLA looser than the SLO. 4) Alert on burn rate with a long and a short window: page on fast burns, ticket on slow ones. 5) Agree in advance that a spent budget freezes feature releases. 6) Write latency SLOs as percentiles, never averages.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Good requests | 98.5% | 100.0% |
| Budget left | 80% | −12% |
| Burn rate | 0.5 x | 0.5 x |
| Booking p99 | 450 ms | 900 ms |
Steps
- Problem The health check is green, riders see errors
Monitoring calls
/healthzon the booking API every 10 seconds. - Fix SLI: count good requests where users meet you
The team takes the number from the load balancer's log instead: every real booking request, and whether it was good.
- Fix SLO: a target, and the budget it leaves
The team agrees on an SLO: 99.9% of booking requests are good over a rolling 30 days.
- Case SLA: the promise with money attached
Corporate clients get an SLA, a contract: 99.5% per month, or the taxi company pays back part of the bill.
- Problem A release burns the budget fifty times too fast
Release v412 adds a fare cache.
- Problem Two naive alerts: too noisy and too late
Rule A pages when the error rate is over 0.1% for 5 minutes.
- Fix Burn-rate alerts with two windows
The page now fires when the burn rate is above 14.4 over the last hour and also over the last 5 minutes.
- Fix Budget spent: releases stop
Later in the month a database migration and a provider outage overspend the budget: 12% over.
- Case A latency SLO is a percentile, not an average
Success is not only "no error".
- Recap Recap
Six rules for SLOs.
Terms
- SLI (service level indicator)
- A measure of what users feel: the share of good requests, or of requests faster than a threshold. For example, "the percentage of answers without error and under 300 ms".
- SLO (service level objective)
- An internal target for an SLI over a window: "99.9% of requests are good over 28 days". Nobody targets 100%: each extra nine costs far more and users would not notice.
- SLA (service level agreement)
- A promise in a contract with a customer: breaking it means a refund or credit. It is set looser than the internal SLO, so the team hears about trouble before the contract is broken.
- Error budget
- The share of bad requests an SLO allows: at 99.9% it is 0.1%, about 43 minutes of full outage in 30 days. While budget is left you ship; when it runs out, releases freeze and reliability work comes first.
- Burn rate
- How many times faster than normal the error budget is being spent: 1 runs out exactly at the end of the window, 14.4 burns 2% of a 30-day budget in one hour. Alerts check a long and a short window together.