mufassall

Cheat sheet · Reliability

SLOs and error budgets

14 min · Intermediate · mufassall.dtwin.uz/en/l/slo-error-budget/

← Open the lesson

The problem

A taxi app's booking API serves about 100 requests a second. The health check says 100% while riders see errors. How to measure reliability the way users feel it, agree on a target, turn the gap into an error budget, and page people only when that budget is really burning.

What you learn

  • SLI, SLO and SLA: what is measured, what is the target, what is promised
  • Error budget math: 99.9% over 30 days is about 43 minutes or 259 200 failed requests
  • Burn rate and multiwindow alerts that page fast and stay quiet on blips
  • Release freezes when the budget is spent, and latency SLOs by percentile
Final scene
40%55%10%45%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_riders(["Riders"])
  n_lb{"Load balancer"}
  n_api["Booking API"]
  n_db[("Trips DB")]
  n_monitor["Monitoring"]
  n_pipeline("Release pipeline")
  n_pager(("On-call"))
  n_riders --> n_lb
  n_lb --> n_api
  n_api --> n_db
  n_lb --> n_monitor
  n_pipeline --> n_api
  n_monitor --> n_pager
  n_pipeline --> n_monitor

Mermaid: request sequence

sequenceDiagram
  participant n_monitor as Monitoring
  participant n_api as Booking API
  participant n_lb as Load balancer
  participant n_riders as Riders
  participant n_pipeline as Release pipeline
  actor n_pager as On-call
  Note over n_monitor,n_api: The health check is green, riders see errors
  n_monitor->>n_api: GET /healthz HTTP/1.1
  n_api-->>n_monitor: HTTP/1.1 200 OK
  Note over n_lb,n_monitor: SLI: count good requests where users meet you
  n_lb->>n_monitor: lb access log, POST /rides, 10:00-10:01
  Note over n_lb,n_monitor: SLO: a target, and the budget it leaves
  n_lb->>n_monitor: slo ride-booking
  Note over n_riders,n_lb: SLA: the promise with money attached
  n_riders->>n_lb: SLA, corporate rides contract
  Note over n_pipeline,n_api: A release burns the budget fifty times too fast
  n_pipeline->>n_api: deploy ride-api v412 to 100%
  n_api-->>n_pipeline: error rate 5.0%, the SLO allows 0.1%
  Note over n_monitor,n_pager: Two naive alerts: too noisy and too late
  n_monitor->>n_pager: rule A: error_rate_5m #62; 0.1% -#62; page
  Note over n_monitor,n_pager: Burn-rate alerts with two windows
  n_monitor->>n_pager: ALERT BudgetBurnFast (page)
  n_pager-->>n_monitor: v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
  Note over n_pipeline,n_monitor: Budget spent: releases stop
  n_pipeline->>n_monitor: GET /slo/ride-booking/budget HTTP/1.1
  n_monitor-->>n_pipeline: HTTP/1.1 200 OK
  Note over n_lb,n_monitor: A latency SLO is a percentile, not an average
  n_lb->>n_monitor: histogram ride_request_duration, last 30 days

D2

direction: right

n_riders: "Riders" {
  shape: rectangle
}
n_lb: "Load balancer" {
  shape: diamond
}
n_api: "Booking API" {
  shape: rectangle
}
n_db: "Trips DB" {
  shape: cylinder
}
n_monitor: "Monitoring" {
  shape: rectangle
}
n_pipeline: "Release pipeline" {
  shape: rectangle
  style.multiple: true
}
n_pager: "On-call" {
  shape: person
}

n_riders -> n_lb
n_lb -> n_api
n_api -> n_db
n_lb -> n_monitor
n_pipeline -> n_api
n_monitor -> n_pager
n_pipeline -> n_monitor

PlantUML: request sequence

@startuml
participant "Monitoring" as n_monitor
participant "Booking API" as n_api
participant "Load balancer" as n_lb
participant "Riders" as n_riders
participant "Release pipeline" as n_pipeline
actor "On-call" as n_pager
note over n_monitor, n_api : The health check is green, riders see errors
n_monitor -> n_api : GET /healthz HTTP/1.1
n_api --> n_monitor : HTTP/1.1 200 OK
note over n_lb, n_monitor : SLI: count good requests where users meet you
n_lb -> n_monitor : lb access log, POST /rides, 10:00-10:01
note over n_lb, n_monitor : SLO: a target, and the budget it leaves
n_lb -> n_monitor : slo ride-booking
note over n_riders, n_lb : SLA: the promise with money attached
n_riders -> n_lb : SLA, corporate rides contract
note over n_pipeline, n_api : A release burns the budget fifty times too fast
n_pipeline -> n_api : deploy ride-api v412 to 100%
n_api --> n_pipeline : error rate 5.0%, the SLO allows 0.1%
note over n_monitor, n_pager : Two naive alerts: too noisy and too late
n_monitor -> n_pager : rule A: error_rate_5m > 0.1% -> page
note over n_monitor, n_pager : Burn-rate alerts with two windows
n_monitor -> n_pager : ALERT BudgetBurnFast (page)
n_pager --> n_monitor : v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
note over n_pipeline, n_monitor : Budget spent: releases stop
n_pipeline -> n_monitor : GET /slo/ride-booking/budget HTTP/1.1
n_monitor --> n_pipeline : HTTP/1.1 200 OK
note over n_lb, n_monitor : A latency SLO is a percentile, not an average
n_lb -> n_monitor : histogram ride_request_duration, last 30 days
@enduml

PlantUML: components

@startuml
left to right direction
agent "Riders" as n_riders
node "Load balancer" as n_lb
component "Booking API" as n_api
database "Trips DB" as n_db
component "Monitoring" as n_monitor
collections "Release pipeline" as n_pipeline
actor "On-call" as n_pager
n_riders --> n_lb
n_lb --> n_api
n_api --> n_db
n_lb --> n_monitor
n_pipeline --> n_api
n_monitor --> n_pager
n_pipeline --> n_monitor
@enduml

Recap

Six rules for SLOs.

1) Measure the SLI where users meet you: good requests over valid ones, at the load balancer. 2) Pick an SLO below 100%; the rest is the error budget (99.9% over 30 days is about 43 minutes). 3) Keep the SLA looser than the SLO. 4) Alert on burn rate with a long and a short window: page on fast burns, ticket on slow ones. 5) Agree in advance that a spent budget freezes feature releases. 6) Write latency SLOs as percentiles, never averages.

Numbers: at the start and at the end

MetricStartEnd
Good requests98.5%100.0%
Budget left80%−12%
Burn rate0.5 x0.5 x
Booking p99450 ms900 ms

Steps

  1. Problem The health check is green, riders see errors

    Monitoring calls /healthz on the booking API every 10 seconds.

  2. Fix SLI: count good requests where users meet you

    The team takes the number from the load balancer's log instead: every real booking request, and whether it was good.

  3. Fix SLO: a target, and the budget it leaves

    The team agrees on an SLO: 99.9% of booking requests are good over a rolling 30 days.

  4. Case SLA: the promise with money attached

    Corporate clients get an SLA, a contract: 99.5% per month, or the taxi company pays back part of the bill.

  5. Problem A release burns the budget fifty times too fast

    Release v412 adds a fare cache.

  6. Problem Two naive alerts: too noisy and too late

    Rule A pages when the error rate is over 0.1% for 5 minutes.

  7. Fix Burn-rate alerts with two windows

    The page now fires when the burn rate is above 14.4 over the last hour and also over the last 5 minutes.

  8. Fix Budget spent: releases stop

    Later in the month a database migration and a provider outage overspend the budget: 12% over.

  9. Case A latency SLO is a percentile, not an average

    Success is not only "no error".

  10. Recap Recap

    Six rules for SLOs.

Terms

SLI (service level indicator)
A measure of what users feel: the share of good requests, or of requests faster than a threshold. For example, "the percentage of answers without error and under 300 ms".
SLO (service level objective)
An internal target for an SLI over a window: "99.9% of requests are good over 28 days". Nobody targets 100%: each extra nine costs far more and users would not notice.
SLA (service level agreement)
A promise in a contract with a customer: breaking it means a refund or credit. It is set looser than the internal SLO, so the team hears about trouble before the contract is broken.
Error budget
The share of bad requests an SLO allows: at 99.9% it is 0.1%, about 43 minutes of full outage in 30 days. While budget is left you ship; when it runs out, releases freeze and reliability work comes first.
Burn rate
How many times faster than normal the error budget is being spent: 1 runs out exactly at the end of the window, 14.4 burns 2% of a 30-day budget in one hour. Alerts check a long and a short window together.