Cheat sheet · Foundations
Latency and throughput
The problem
How long one request takes and how many requests fit into a second are two different numbers. A search API grows from a handful of users to a rush hour: why the average hides the pain, and why a server at 90% is already in trouble.
What you learn
- Latency vs throughput, and how they are linked by the number of requests in flight
- Reading p50, p95 and p99, and why the average lies
- Why waiting time explodes as a server nears 100% load
- Shedding load early and taking slow work off the request path
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_users(["Users"])
n_api["Search API"]
n_db[("Database")]
n_counter{{"Counter cache"}}
n_users --> n_api
n_api -.-> n_users
n_api --> n_db
n_api --> n_counterMermaid: request sequence
sequenceDiagram
participant n_users as Users
participant n_api as Search API
participant n_db as Database
participant n_counter as Counter cache
Note over n_users,n_api: Latency: how long one request takes
n_users->>n_api: GET /search?q=lamp HTTP/1.1
n_api-->>n_users: HTTP/1.1 200 OK
Note over n_users,n_api: Throughput: how many per second
n_users->>n_api: load test: 60 s
Note over n_users,n_api: The average says everything is fine
n_users->>n_api: access log, last 10000 requests
Note over n_api,n_db: Where the tail comes from
n_api->>n_db: UPDATE search_stats SET hits = hits + 1
n_db-->>n_api: UPDATE 1
Note over n_users,n_api: Rush hour: the server is 93% busy
n_users->>n_api: GET /search?q=lamp HTTP/1.1
n_api-->>n_users: HTTP/1.1 200 OK
Note over n_users,n_api: Clients give up and try again
n_users->>n_api: GET /search?q=lamp
Note over n_users,n_api: Say no quickly
n_users->>n_api: GET /search?q=lamp HTTP/1.1
n_api-->>n_users: HTTP/1.1 503 Service Unavailable
Note over n_api,n_counter: Take slow work off the request path
n_api->>n_counter: INCR search:lamp:hits
n_counter-->>n_api: (integer) 51207D2
direction: right
n_users: "Users" {
shape: rectangle
}
n_api: "Search API" {
shape: rectangle
}
n_db: "Database" {
shape: cylinder
}
n_counter: "Counter cache" {
shape: hexagon
}
n_users -> n_api
n_api -> n_users {
style.stroke-dash: 3
}
n_api -> n_db
n_api -> n_counterPlantUML: request sequence
@startuml
participant "Users" as n_users
participant "Search API" as n_api
database "Database" as n_db
collections "Counter cache" as n_counter
note over n_users, n_api : Latency: how long one request takes
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 200 OK
note over n_users, n_api : Throughput: how many per second
n_users -> n_api : load test: 60 s
note over n_users, n_api : The average says everything is fine
n_users -> n_api : access log, last 10000 requests
note over n_api, n_db : Where the tail comes from
n_api -> n_db : UPDATE search_stats SET hits = hits + 1
n_db --> n_api : UPDATE 1
note over n_users, n_api : Rush hour: the server is 93% busy
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 200 OK
note over n_users, n_api : Clients give up and try again
n_users -> n_api : [retry] GET /search?q=lamp
note over n_users, n_api : Say no quickly
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 503 Service Unavailable
note over n_api, n_counter : Take slow work off the request path
n_api -> n_counter : INCR search:lamp:hits
n_counter --> n_api : (integer) 51207
@endumlPlantUML: components
@startuml
left to right direction
agent "Users" as n_users
component "Search API" as n_api
database "Database" as n_db
stack "Counter cache" as n_counter
n_users --> n_api
n_api ..> n_users
n_api --> n_db
n_api --> n_counter
@endumlRecap
Latency is the time of one request; throughput is how many finish per second. They meet through the number of requests in flight.
1) Watch p95 and p99, not the average. 2) Near full load, waiting, not work, dominates latency, so keep headroom. 3) Retries need backoff; overload needs a fast 503. 4) Shorten the request path and the tail shrinks with it.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Throughput | 5/s | 450/s |
| Latency p50 | 41 ms | 22 ms |
| Latency p99 | 60 ms | 85 ms |
| Errors | 0% | 0% |
Steps
- Case Latency: how long one request takes
A user searches for “lamp”.
- Case Throughput: how many per second
Now 200 requests arrive every second, and the API answers all of them.
- Problem The average says everything is fine
The dashboard shows an average of 62 ms.
- Problem Where the tail comes from
The slow requests have something in common: every search also increments a counter for its query.
- Problem Rush hour: the server is 93% busy
In the evening traffic more than doubles, to 450 requests per second.
- Case Why 90% is already too much
Think of a cashier who needs 1 minute per customer.
- Problem Clients give up and try again
The mobile app gives up after 3 s and immediately sends the same search again.
- Fix Say no quickly
The API now caps the line: at most 64 requests in flight and 100 waiting.
- Fix Take slow work off the request path
The search counter no longer touches the database on every request.
- Recap Recap
Latency is the time of one request; throughput is how many finish per second.
Terms
- Latency
- How long one request takes from start to finish. Under load, time spent waiting in line is added to the work itself.
- Throughput
- How many requests a system finishes per second. It is linked to latency through the number of requests in flight.
- Percentile (p50, p95, p99)
- p95 means 95% of requests are faster than this value, so one in twenty is slower. The average hides the slow tail, so you watch p95 and p99.
- Load shedding
- When a server is full, extra requests get an instant 503 instead of waiting. A fast "try again" for a few beats a slow failure for everyone.