mufassall

Cheat sheet · Foundations

Latency and throughput

11 min · Beginner · mufassall.dtwin.uz/en/l/latency-throughput/

← Open the lesson

The problem

How long one request takes and how many requests fit into a second are two different numbers. A search API grows from a handful of users to a rush hour: why the average hides the pain, and why a server at 90% is already in trouble.

What you learn

  • Latency vs throughput, and how they are linked by the number of requests in flight
  • Reading p50, p95 and p99, and why the average lies
  • Why waiting time explodes as a server nears 100% load
  • Shedding load early and taking slow work off the request path
Final scene
48%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_users(["Users"])
  n_api["Search API"]
  n_db[("Database")]
  n_counter{{"Counter cache"}}
  n_users --> n_api
  n_api -.-> n_users
  n_api --> n_db
  n_api --> n_counter

Mermaid: request sequence

sequenceDiagram
  participant n_users as Users
  participant n_api as Search API
  participant n_db as Database
  participant n_counter as Counter cache
  Note over n_users,n_api: Latency: how long one request takes
  n_users->>n_api: GET /search?q=lamp HTTP/1.1
  n_api-->>n_users: HTTP/1.1 200 OK
  Note over n_users,n_api: Throughput: how many per second
  n_users->>n_api: load test: 60 s
  Note over n_users,n_api: The average says everything is fine
  n_users->>n_api: access log, last 10000 requests
  Note over n_api,n_db: Where the tail comes from
  n_api->>n_db: UPDATE search_stats SET hits = hits + 1
  n_db-->>n_api: UPDATE 1
  Note over n_users,n_api: Rush hour: the server is 93% busy
  n_users->>n_api: GET /search?q=lamp HTTP/1.1
  n_api-->>n_users: HTTP/1.1 200 OK
  Note over n_users,n_api: Clients give up and try again
  n_users->>n_api: GET /search?q=lamp
  Note over n_users,n_api: Say no quickly
  n_users->>n_api: GET /search?q=lamp HTTP/1.1
  n_api-->>n_users: HTTP/1.1 503 Service Unavailable
  Note over n_api,n_counter: Take slow work off the request path
  n_api->>n_counter: INCR search:lamp:hits
  n_counter-->>n_api: (integer) 51207

D2

direction: right

n_users: "Users" {
  shape: rectangle
}
n_api: "Search API" {
  shape: rectangle
}
n_db: "Database" {
  shape: cylinder
}
n_counter: "Counter cache" {
  shape: hexagon
}

n_users -> n_api
n_api -> n_users {
  style.stroke-dash: 3
}
n_api -> n_db
n_api -> n_counter

PlantUML: request sequence

@startuml
participant "Users" as n_users
participant "Search API" as n_api
database "Database" as n_db
collections "Counter cache" as n_counter
note over n_users, n_api : Latency: how long one request takes
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 200 OK
note over n_users, n_api : Throughput: how many per second
n_users -> n_api : load test: 60 s
note over n_users, n_api : The average says everything is fine
n_users -> n_api : access log, last 10000 requests
note over n_api, n_db : Where the tail comes from
n_api -> n_db : UPDATE search_stats SET hits = hits + 1
n_db --> n_api : UPDATE 1
note over n_users, n_api : Rush hour: the server is 93% busy
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 200 OK
note over n_users, n_api : Clients give up and try again
n_users -> n_api : [retry] GET /search?q=lamp
note over n_users, n_api : Say no quickly
n_users -> n_api : GET /search?q=lamp HTTP/1.1
n_api --> n_users : HTTP/1.1 503 Service Unavailable
note over n_api, n_counter : Take slow work off the request path
n_api -> n_counter : INCR search:lamp:hits
n_counter --> n_api : (integer) 51207
@enduml

PlantUML: components

@startuml
left to right direction
agent "Users" as n_users
component "Search API" as n_api
database "Database" as n_db
stack "Counter cache" as n_counter
n_users --> n_api
n_api ..> n_users
n_api --> n_db
n_api --> n_counter
@enduml

Recap

Latency is the time of one request; throughput is how many finish per second. They meet through the number of requests in flight.

1) Watch p95 and p99, not the average. 2) Near full load, waiting, not work, dominates latency, so keep headroom. 3) Retries need backoff; overload needs a fast 503. 4) Shorten the request path and the tail shrinks with it.

Numbers: at the start and at the end

MetricStartEnd
Throughput5/s450/s
Latency p5041 ms22 ms
Latency p9960 ms85 ms
Errors0%0%

Steps

  1. Case Latency: how long one request takes

    A user searches for “lamp”.

  2. Case Throughput: how many per second

    Now 200 requests arrive every second, and the API answers all of them.

  3. Problem The average says everything is fine

    The dashboard shows an average of 62 ms.

  4. Problem Where the tail comes from

    The slow requests have something in common: every search also increments a counter for its query.

  5. Problem Rush hour: the server is 93% busy

    In the evening traffic more than doubles, to 450 requests per second.

  6. Case Why 90% is already too much

    Think of a cashier who needs 1 minute per customer.

  7. Problem Clients give up and try again

    The mobile app gives up after 3 s and immediately sends the same search again.

  8. Fix Say no quickly

    The API now caps the line: at most 64 requests in flight and 100 waiting.

  9. Fix Take slow work off the request path

    The search counter no longer touches the database on every request.

  10. Recap Recap

    Latency is the time of one request; throughput is how many finish per second.

Terms

Latency
How long one request takes from start to finish. Under load, time spent waiting in line is added to the work itself.
Throughput
How many requests a system finishes per second. It is linked to latency through the number of requests in flight.
Percentile (p50, p95, p99)
p95 means 95% of requests are faster than this value, so one in twenty is slower. The average hides the slow tail, so you watch p95 and p99.
Load shedding
When a server is full, extra requests get an instant 503 instead of waiting. A fast "try again" for a few beats a slow failure for everyone.