Cheat sheet · Communication
Message queues
The problem
A shop sends an order email through a provider that takes 100 messages a second, and a sale brings 600 orders a second. How a queue absorbs the spike, and what you must set on it: leases, acks, backoff, a dead-letter queue and backpressure.
What you learn
- How a queue between producer and workers smooths a spike
- Visibility timeout and ack: what happens when a worker dies or is slow
- Retries with exponential backoff and jitter, and a dead-letter queue for poison messages
- Scaling consumers by backlog and slowing producers with backpressure
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_buyers(["Buyers"])
n_api["Shop API"]
n_mail>"Mail provider"]
n_queue[["Queue"]]
n_w1("Worker 1")
n_w2("Worker 2")
n_dlq[["Dead-letter queue"]]
n_campaign("Newsletter job")
n_buyers -->|"call"| n_api
n_api -->|"async"| n_queue
n_queue -->|"async"| n_w1
n_w1 -->|"call"| n_mail
n_queue -->|"async"| n_w2
n_w2 -->|"call"| n_mail
n_queue --x|"async"| n_dlq
n_campaign -->|"async"| n_queueMermaid: request sequence
sequenceDiagram
participant n_api as Shop API
participant n_mail as Mail provider
participant n_queue as Queue
participant n_w2 as Worker 2
participant n_w1 as Worker 1
participant n_dlq as Dead-letter queue
participant n_campaign as Newsletter job
Note over n_api,n_mail: The email is sent inside the order request
n_api->>n_mail: POST /v1/send HTTP/1.1
n_mail-->>n_api: HTTP/1.1 202 Accepted
Note over n_api,n_mail: A sale: 600 orders a second
n_api->>n_mail: POST /v1/send HTTP/1.1
n_mail-->>n_api: HTTP/1.1 429 Too Many Requests
Note over n_api,n_queue: A queue between the order and the email
n_api->>n_queue: SEND emails
n_queue-->>n_api: OK queued, depth 42000
Note over n_queue,n_w2: More workers, up to the provider's cap
n_queue->>n_w2: autoscaler: queue emails
Note over n_queue,n_w1: A worker dies holding a message
n_queue->>n_w1: RECEIVE emails
n_w1-->>n_queue: DELETE m_8812
Note over n_queue,n_w2: A slow job outlives its lease
n_queue->>n_w2: 10:02:00 w1 got m_9120 (receipt: 212 pages)
Note over n_queue,n_w1: Extend the lease while working
n_queue->>n_w1: EXTEND emails m_9120 visibility_timeout=60s
n_w1-->>n_queue: OK until 10:03:20
Note over n_w1,n_mail: The provider is down, workers hammer it
n_w1->>n_mail: POST /v1/send HTTP/1.1
n_mail-->>n_w1: HTTP/1.1 503 Service Unavailable
Note over n_queue,n_w1: Exponential backoff with jitter
n_queue->>n_w1: NACK emails m_8812 delay=86s
Note over n_queue,n_dlq: A poison message goes to the dead-letter queue
n_queue->>n_dlq: SET emails redrive_policy
n_dlq-->>n_queue: m_7731 moved to emails-dlq after 5 receives
Note over n_campaign,n_queue: Backpressure: the producer slows down
n_campaign->>n_queue: campaign: 2000000 newsletter emails to publishD2
direction: right
n_buyers: "Buyers" {
shape: rectangle
}
n_api: "Shop API" {
shape: rectangle
}
n_mail: "Mail provider" {
shape: cloud
}
n_queue: "Queue" {
shape: queue
}
n_w1: "Worker 1" {
shape: rectangle
style.multiple: true
}
n_w2: "Worker 2" {
shape: rectangle
style.multiple: true
}
n_dlq: "Dead-letter queue" {
shape: queue
}
n_campaign: "Newsletter job" {
shape: rectangle
style.multiple: true
}
n_buyers -> n_api: "call"
n_api -> n_queue: "async"
n_queue -> n_w1: "async"
n_w1 -> n_mail: "call"
n_queue -> n_w2: "async"
n_w2 -> n_mail: "call"
n_queue -> n_dlq: "async" {
target-arrowhead.shape: cross
}
n_campaign -> n_queue: "async"PlantUML: request sequence
@startuml
participant "Shop API" as n_api
boundary "Mail provider" as n_mail
queue "Queue" as n_queue
participant "Worker 2" as n_w2
participant "Worker 1" as n_w1
queue "Dead-letter queue" as n_dlq
participant "Newsletter job" as n_campaign
note over n_api, n_mail : The email is sent inside the order request
n_api -> n_mail : POST /v1/send HTTP/1.1
n_mail --> n_api : HTTP/1.1 202 Accepted
note over n_api, n_mail : A sale: 600 orders a second
n_api -> n_mail : POST /v1/send HTTP/1.1
n_mail --> n_api : HTTP/1.1 429 Too Many Requests
note over n_api, n_queue : A queue between the order and the email
n_api ->> n_queue : SEND emails
n_queue -->> n_api : OK queued, depth 42000
note over n_queue, n_w2 : More workers, up to the provider's cap
n_queue ->> n_w2 : autoscaler: queue emails
note over n_queue, n_w1 : A worker dies holding a message
n_queue ->> n_w1 : RECEIVE emails
n_w1 -->> n_queue : DELETE m_8812
note over n_queue, n_w2 : A slow job outlives its lease
n_queue ->> n_w2 : [duplicate] 10:02:00 w1 got m_9120 (receipt: 212 pages)
note over n_queue, n_w1 : Extend the lease while working
n_queue ->> n_w1 : EXTEND emails m_9120 visibility_timeout=60s
n_w1 -->> n_queue : OK until 10:03:20
note over n_w1, n_mail : The provider is down, workers hammer it
n_w1 -> n_mail : [retry] POST /v1/send HTTP/1.1
n_mail --> n_w1 : HTTP/1.1 503 Service Unavailable
note over n_queue, n_w1 : Exponential backoff with jitter
n_queue ->> n_w1 : NACK emails m_8812 delay=86s
note over n_queue, n_dlq : A poison message goes to the dead-letter queue
n_queue ->> n_dlq : SET emails redrive_policy
n_dlq -->> n_queue : m_7731 moved to emails-dlq after 5 receives
note over n_campaign, n_queue : Backpressure: the producer slows down
n_campaign ->> n_queue : campaign: 2000000 newsletter emails to publish
@endumlPlantUML: components
@startuml
left to right direction
agent "Buyers" as n_buyers
component "Shop API" as n_api
cloud "Mail provider" as n_mail
queue "Queue" as n_queue
collections "Worker 1" as n_w1
collections "Worker 2" as n_w2
queue "Dead-letter queue" as n_dlq
collections "Newsletter job" as n_campaign
n_buyers --> n_api : call
n_api --> n_queue : async
n_queue --> n_w1 : async
n_w1 --> n_mail : call
n_queue --> n_w2 : async
n_w2 --> n_mail : call
n_queue -[#c62828]-> n_dlq : async
n_campaign --> n_queue : async
@endumlRecap
Six rules for a queue.
1) Slow and limited work leaves the request path and goes through a queue. 2) Scale workers by backlog and the oldest message's age, up to the downstream limit. 3) Ack after the work; extend the lease for long jobs. 4) Retry with exponential backoff and jitter. 5) Poison messages go to a watched dead-letter queue. 6) Bound the queue and let producers slow down.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Order p95 | 950 ms | 35 ms |
| Backlog | 0 | 20 000 |
| Email delay | 1 s | 60 s |
| Lost emails | 0/h | 0/h |
Steps
- Case The email is sent inside the order request
When an order is paid, the API calls the mail provider and waits for its answer before replying to the buyer.
- Problem A sale: 600 orders a second
At 10:00 a sale starts and orders jump fifteen times.
- Fix A queue between the order and the email
The API now only puts a small job in a queue and answers the buyer at once.
- Fix More workers, up to the provider's cap
The autoscaler watches the backlog and the age of the oldest message, and starts a second worker.
- Case A worker dies holding a message
Worker 2 takes message
m_8812and dies before sending it. - Problem A slow job outlives its lease
A wholesale order has a 212-page receipt, and the PDF takes 45 seconds.
- Fix Extend the lease while working
While a job runs, the worker extends its lease every 20 seconds.
- Problem The provider is down, workers hammer it
The mail provider goes down for twenty minutes.
- Fix Exponential backoff with jitter
After a failure the worker returns the message with a delay that doubles each time: 10 s, 20 s, 40 s, 80 s, capped at 15 minutes.
- Fix A poison message goes to the dead-letter queue
One message refers to an email template that was deleted.
- Fix Backpressure: the producer slows down
Marketing launches a newsletter: 2 million emails pushed into the same queue at once.
- Recap Recap
Six rules for a queue.
Terms
- Message queue
- A buffer between a producer and workers: slow or limited work leaves the request path and goes into the queue, and workers process it at their own pace. The queue absorbs spikes; ack comes after the work.
- Dead-letter queue
- A separate queue that receives a poison message after it fails several attempts. It stops blocking the main queue and is watched by people.
- Visibility timeout (lease)
- When a queue hands a message to a worker it does not delete it, it hides it from others for this long. If the worker does not ack in time, the message becomes visible again and goes to another worker.