mufassall

Cheat sheet · Reliability

Backpressure

14 min · Intermediate · mufassall.dtwin.uz/en/l/backpressure/

← Open the lesson

The problem

A mobile app sends analytics events to an ingest API, which keeps them in memory until a writer stores them. When the database slows down, events arrive faster than they leave, the buffer has no limit, and the API dies of memory with 1.8 million events already confirmed. Backpressure is how a system says "slower" back up the chain: a bounded buffer, a chosen way to overflow, 429 with Retry-After, consumers that pull, shedding by priority and a limit at the door.

What you learn

  • Why an unbounded queue turns overload into latency and then into lost data, and how to compute it
  • A bounded buffer and the four overflow strategies: block, drop newest, drop oldest, sample
  • Load shedding with 429 and Retry-After, and pull-based flow: consumer poll, TCP window, request(n)
  • Carrying the signal upstream by priority, and a concurrency limit that rejects fast instead of queueing
Final scene
60%70%20%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_apps(["Mobile apps"])
  n_api["Ingest API"]
  n_writer("Writers")
  n_db[("Analytics DB")]
  n_log[["Event log"]]
  n_apps --> n_api
  n_writer --> n_db
  n_api --> n_log
  n_writer --> n_log

Mermaid: request sequence

sequenceDiagram
  participant n_apps as Mobile apps
  participant n_api as Ingest API
  participant n_buffer as Memory buffer
  participant n_writer as Writers
  participant n_log as Event log
  Note over n_apps,n_api: Events go into memory, a writer drains them
  n_apps->>n_api: POST /v1/events HTTP/1.1
  n_api-->>n_apps: HTTP/1.1 202 Accepted
  Note over n_api,n_buffer: The database slows down, the buffer grows
  n_api->>n_buffer: 19:10 buffer in 1 200/s out 800/s
  Note over n_api,n_buffer: The API dies with 1.8 million confirmed events
  n_api->>n_buffer: 20:16:41 api-3 OOMKilled (limit 2 GiB)
  Note over n_api,n_buffer: The buffer gets a limit: block when full
  n_api->>n_buffer: buffer = BoundedQueue(capacity=50_000)
  Note over n_api,n_buffer: When it is full: block, drop or sample
  n_api->>n_buffer: buffer full (50 000): what happens to event 50 001
  Note over n_apps,n_api: Shed load: a fast no with Retry-After
  n_apps->>n_api: POST /v1/events HTTP/1.1
  n_api-->>n_apps: HTTP/1.1 429 Too Many Requests
  Note over n_writer,n_log: A durable log, and writers that pull
  n_writer->>n_log: writer-2 poll(max_records=500)
  Note over n_api,n_log: The lag travels up the chain, by priority
  n_api->>n_log: api admission consumer_lag=74 s (limit 60 s)
  Note over n_apps,n_api: A concurrency limit at the door
  n_apps->>n_api: api concurrency limiter window 1 s

D2

direction: right

n_apps: "Mobile apps" {
  shape: rectangle
}
n_api: "Ingest API" {
  shape: rectangle
}
n_writer: "Writers" {
  shape: rectangle
  style.multiple: true
}
n_db: "Analytics DB" {
  shape: cylinder
}
n_log: "Event log" {
  shape: queue
}

n_apps -> n_api
n_writer -> n_db
n_api -> n_log
n_writer -> n_log

PlantUML: request sequence

@startuml
participant "Mobile apps" as n_apps
participant "Ingest API" as n_api
queue "Memory buffer" as n_buffer
participant "Writers" as n_writer
queue "Event log" as n_log
note over n_apps, n_api : Events go into memory, a writer drains them
n_apps -> n_api : POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 202 Accepted
note over n_api, n_buffer : The database slows down, the buffer grows
n_api -> n_buffer : 19:10 buffer in 1 200/s out 800/s
note over n_api, n_buffer : The API dies with 1.8 million confirmed events
n_api -> n_buffer : 20:16:41 api-3 OOMKilled (limit 2 GiB)
note over n_api, n_buffer : The buffer gets a limit: block when full
n_api -> n_buffer : buffer = BoundedQueue(capacity=50_000)
note over n_api, n_buffer : When it is full: block, drop or sample
n_api -> n_buffer : buffer full (50 000): what happens to event 50 001
note over n_apps, n_api : Shed load: a fast no with Retry-After
n_apps -> n_api : [retry] POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 429 Too Many Requests
note over n_writer, n_log : A durable log, and writers that pull
n_writer -> n_log : writer-2 poll(max_records=500)
note over n_api, n_log : The lag travels up the chain, by priority
n_api -> n_log : api admission consumer_lag=74 s (limit 60 s)
note over n_apps, n_api : A concurrency limit at the door
n_apps -> n_api : api concurrency limiter window 1 s
@enduml

PlantUML: components

@startuml
left to right direction
agent "Mobile apps" as n_apps
component "Ingest API" as n_api
collections "Writers" as n_writer
database "Analytics DB" as n_db
queue "Event log" as n_log
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_log
@enduml

Recap

Six rules for backpressure.

1) If load is above 1, the queue grows by (in − out) every second: compute it, do not hope. 2) Every buffer has a limit, chosen by the delay you accept. 3) Decide what happens when it is full: block, drop newest, drop oldest or sample, and count what you drop. 4) Shed with a fast 429 and Retry-After; clients keep data and retry with jitter. 5) Let consumers pull, keep the backlog in a durable log and alert on consumer lag. 6) Pass the signal upstream by priority and limit concurrency at the door, so rejection is cheap.

Numbers: at the start and at the end

MetricStartEnd
API memory40 MB60 MB
Event delay0.5 s1 s
Events lost00
Events turned away0%0%

Steps

  1. Case Events go into memory, a writer drains them

    The app's SDK collects events and sends them in batches of up to 50.

  2. Problem The database slows down, the buffer grows

    At 19:00 a nightly index rebuild starts and the writers drop to 800 events a second.

  3. Problem The API dies with 1.8 million confirmed events

    At 20:16 the API container reaches its 2 GiB limit and is killed.

  4. Fix The buffer gets a limit: block when full

    The buffer now holds at most 50,000 events: about 50 MB, never more.

  5. Fix When it is full: block, drop or sample

    Blocking is one answer, not the only one.

  6. Fix Shed load: a fast no with Retry-After

    Blocked requests still hold a connection and a thread.

  7. Fix A durable log, and writers that pull

    The memory buffer is replaced by an event log on disk with three replicas.

  8. Fix The lag travels up the chain, by priority

    The API reads the consumer lag every 5 seconds.

  9. Fix A concurrency limit at the door

    The last guard is at the API itself.

  10. Recap Recap

    Six rules for backpressure.

Terms

Backpressure
Slowing the producer down when consumers cannot keep up. The queue is bounded, and the producer waits instead of the queue growing without limit.
Bounded queue
A queue with a fixed capacity. What happens when it is full (wait, drop the newest or the oldest) is an explicit rule, and memory never grows without limit.
Consumer lag
How far a consumer is behind the end of the log, in records or seconds. The most honest overload signal: alert on it, not on CPU.
Retry-After
A header on a 429 or 503 response: how many seconds until the client should come back. The client waits that long plus random jitter and keeps its data.
Admission control
Deciding at the door which requests to take and which to reject at once, by a limit on concurrent requests and by priority. A fast rejection is cheap, waiting in a queue is not.
Pull-based flow
The consumer asks for the next batch only after finishing the last one (poll, request(n), the TCP window). The receiver sets the pace, not the producer.