Cheat sheet · Reliability
Backpressure
The problem
A mobile app sends analytics events to an ingest API, which keeps them in memory until a writer stores them. When the database slows down, events arrive faster than they leave, the buffer has no limit, and the API dies of memory with 1.8 million events already confirmed. Backpressure is how a system says "slower" back up the chain: a bounded buffer, a chosen way to overflow, 429 with Retry-After, consumers that pull, shedding by priority and a limit at the door.
What you learn
- Why an unbounded queue turns overload into latency and then into lost data, and how to compute it
- A bounded buffer and the four overflow strategies: block, drop newest, drop oldest, sample
- Load shedding with 429 and Retry-After, and pull-based flow: consumer poll, TCP window, request(n)
- Carrying the signal upstream by priority, and a concurrency limit that rejects fast instead of queueing
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_apps(["Mobile apps"])
n_api["Ingest API"]
n_writer("Writers")
n_db[("Analytics DB")]
n_log[["Event log"]]
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_logMermaid: request sequence
sequenceDiagram
participant n_apps as Mobile apps
participant n_api as Ingest API
participant n_buffer as Memory buffer
participant n_writer as Writers
participant n_log as Event log
Note over n_apps,n_api: Events go into memory, a writer drains them
n_apps->>n_api: POST /v1/events HTTP/1.1
n_api-->>n_apps: HTTP/1.1 202 Accepted
Note over n_api,n_buffer: The database slows down, the buffer grows
n_api->>n_buffer: 19:10 buffer in 1 200/s out 800/s
Note over n_api,n_buffer: The API dies with 1.8 million confirmed events
n_api->>n_buffer: 20:16:41 api-3 OOMKilled (limit 2 GiB)
Note over n_api,n_buffer: The buffer gets a limit: block when full
n_api->>n_buffer: buffer = BoundedQueue(capacity=50_000)
Note over n_api,n_buffer: When it is full: block, drop or sample
n_api->>n_buffer: buffer full (50 000): what happens to event 50 001
Note over n_apps,n_api: Shed load: a fast no with Retry-After
n_apps->>n_api: POST /v1/events HTTP/1.1
n_api-->>n_apps: HTTP/1.1 429 Too Many Requests
Note over n_writer,n_log: A durable log, and writers that pull
n_writer->>n_log: writer-2 poll(max_records=500)
Note over n_api,n_log: The lag travels up the chain, by priority
n_api->>n_log: api admission consumer_lag=74 s (limit 60 s)
Note over n_apps,n_api: A concurrency limit at the door
n_apps->>n_api: api concurrency limiter window 1 sD2
direction: right
n_apps: "Mobile apps" {
shape: rectangle
}
n_api: "Ingest API" {
shape: rectangle
}
n_writer: "Writers" {
shape: rectangle
style.multiple: true
}
n_db: "Analytics DB" {
shape: cylinder
}
n_log: "Event log" {
shape: queue
}
n_apps -> n_api
n_writer -> n_db
n_api -> n_log
n_writer -> n_logPlantUML: request sequence
@startuml
participant "Mobile apps" as n_apps
participant "Ingest API" as n_api
queue "Memory buffer" as n_buffer
participant "Writers" as n_writer
queue "Event log" as n_log
note over n_apps, n_api : Events go into memory, a writer drains them
n_apps -> n_api : POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 202 Accepted
note over n_api, n_buffer : The database slows down, the buffer grows
n_api -> n_buffer : 19:10 buffer in 1 200/s out 800/s
note over n_api, n_buffer : The API dies with 1.8 million confirmed events
n_api -> n_buffer : 20:16:41 api-3 OOMKilled (limit 2 GiB)
note over n_api, n_buffer : The buffer gets a limit: block when full
n_api -> n_buffer : buffer = BoundedQueue(capacity=50_000)
note over n_api, n_buffer : When it is full: block, drop or sample
n_api -> n_buffer : buffer full (50 000): what happens to event 50 001
note over n_apps, n_api : Shed load: a fast no with Retry-After
n_apps -> n_api : [retry] POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 429 Too Many Requests
note over n_writer, n_log : A durable log, and writers that pull
n_writer -> n_log : writer-2 poll(max_records=500)
note over n_api, n_log : The lag travels up the chain, by priority
n_api -> n_log : api admission consumer_lag=74 s (limit 60 s)
note over n_apps, n_api : A concurrency limit at the door
n_apps -> n_api : api concurrency limiter window 1 s
@endumlPlantUML: components
@startuml
left to right direction
agent "Mobile apps" as n_apps
component "Ingest API" as n_api
collections "Writers" as n_writer
database "Analytics DB" as n_db
queue "Event log" as n_log
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_log
@endumlRecap
Six rules for backpressure.
1) If load is above 1, the queue grows by (in − out) every second: compute it, do not hope. 2) Every buffer has a limit, chosen by the delay you accept. 3) Decide what happens when it is full: block, drop newest, drop oldest or sample, and count what you drop. 4) Shed with a fast 429 and Retry-After; clients keep data and retry with jitter. 5) Let consumers pull, keep the backlog in a durable log and alert on consumer lag. 6) Pass the signal upstream by priority and limit concurrency at the door, so rejection is cheap.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| API memory | 40 MB | 60 MB |
| Event delay | 0.5 s | 1 s |
| Events lost | 0 | 0 |
| Events turned away | 0% | 0% |
Steps
- Case Events go into memory, a writer drains them
The app's SDK collects events and sends them in batches of up to 50.
- Problem The database slows down, the buffer grows
At 19:00 a nightly index rebuild starts and the writers drop to 800 events a second.
- Problem The API dies with 1.8 million confirmed events
At 20:16 the API container reaches its 2 GiB limit and is killed.
- Fix The buffer gets a limit: block when full
The buffer now holds at most 50,000 events: about 50 MB, never more.
- Fix When it is full: block, drop or sample
Blocking is one answer, not the only one.
- Fix Shed load: a fast no with Retry-After
Blocked requests still hold a connection and a thread.
- Fix A durable log, and writers that pull
The memory buffer is replaced by an event log on disk with three replicas.
- Fix The lag travels up the chain, by priority
The API reads the consumer lag every 5 seconds.
- Fix A concurrency limit at the door
The last guard is at the API itself.
- Recap Recap
Six rules for backpressure.
Terms
- Backpressure
- Slowing the producer down when consumers cannot keep up. The queue is bounded, and the producer waits instead of the queue growing without limit.
- Bounded queue
- A queue with a fixed capacity. What happens when it is full (wait, drop the newest or the oldest) is an explicit rule, and memory never grows without limit.
- Consumer lag
- How far a consumer is behind the end of the log, in records or seconds. The most honest overload signal: alert on it, not on CPU.
- Retry-After
- A header on a 429 or 503 response: how many seconds until the client should come back. The client waits that long plus random jitter and keeps its data.
- Admission control
- Deciding at the door which requests to take and which to reject at once, by a limit on concurrent requests and by priority. A fast rejection is cheap, waiting in a queue is not.
- Pull-based flow
- The consumer asks for the next batch only after finishing the last one (poll, request(n), the TCP window). The receiver sets the pace, not the producer.