mufassall

Cheat sheet · Communication

Message queues

14 min · Intermediate · mufassall.dtwin.uz/en/l/queues/

← Open the lesson

The problem

A shop sends an order email through a provider that takes 100 messages a second, and a sale brings 600 orders a second. How a queue absorbs the spike, and what you must set on it: leases, acks, backoff, a dead-letter queue and backpressure.

What you learn

  • How a queue between producer and workers smooths a spike
  • Visibility timeout and ack: what happens when a worker dies or is slow
  • Retries with exponential backoff and jitter, and a dead-letter queue for poison messages
  • Scaling consumers by backlog and slowing producers with backpressure
Final scene
20%30%50%40%40%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_buyers(["Buyers"])
  n_api["Shop API"]
  n_mail>"Mail provider"]
  n_queue[["Queue"]]
  n_w1("Worker 1")
  n_w2("Worker 2")
  n_dlq[["Dead-letter queue"]]
  n_campaign("Newsletter job")
  n_buyers -->|"call"| n_api
  n_api -->|"async"| n_queue
  n_queue -->|"async"| n_w1
  n_w1 -->|"call"| n_mail
  n_queue -->|"async"| n_w2
  n_w2 -->|"call"| n_mail
  n_queue --x|"async"| n_dlq
  n_campaign -->|"async"| n_queue

Mermaid: request sequence

sequenceDiagram
  participant n_api as Shop API
  participant n_mail as Mail provider
  participant n_queue as Queue
  participant n_w2 as Worker 2
  participant n_w1 as Worker 1
  participant n_dlq as Dead-letter queue
  participant n_campaign as Newsletter job
  Note over n_api,n_mail: The email is sent inside the order request
  n_api->>n_mail: POST /v1/send HTTP/1.1
  n_mail-->>n_api: HTTP/1.1 202 Accepted
  Note over n_api,n_mail: A sale: 600 orders a second
  n_api->>n_mail: POST /v1/send HTTP/1.1
  n_mail-->>n_api: HTTP/1.1 429 Too Many Requests
  Note over n_api,n_queue: A queue between the order and the email
  n_api->>n_queue: SEND emails
  n_queue-->>n_api: OK queued, depth 42000
  Note over n_queue,n_w2: More workers, up to the provider's cap
  n_queue->>n_w2: autoscaler: queue emails
  Note over n_queue,n_w1: A worker dies holding a message
  n_queue->>n_w1: RECEIVE emails
  n_w1-->>n_queue: DELETE m_8812
  Note over n_queue,n_w2: A slow job outlives its lease
  n_queue->>n_w2: 10:02:00 w1 got m_9120 (receipt: 212 pages)
  Note over n_queue,n_w1: Extend the lease while working
  n_queue->>n_w1: EXTEND emails m_9120 visibility_timeout=60s
  n_w1-->>n_queue: OK until 10:03:20
  Note over n_w1,n_mail: The provider is down, workers hammer it
  n_w1->>n_mail: POST /v1/send HTTP/1.1
  n_mail-->>n_w1: HTTP/1.1 503 Service Unavailable
  Note over n_queue,n_w1: Exponential backoff with jitter
  n_queue->>n_w1: NACK emails m_8812 delay=86s
  Note over n_queue,n_dlq: A poison message goes to the dead-letter queue
  n_queue->>n_dlq: SET emails redrive_policy
  n_dlq-->>n_queue: m_7731 moved to emails-dlq after 5 receives
  Note over n_campaign,n_queue: Backpressure: the producer slows down
  n_campaign->>n_queue: campaign: 2000000 newsletter emails to publish

D2

direction: right

n_buyers: "Buyers" {
  shape: rectangle
}
n_api: "Shop API" {
  shape: rectangle
}
n_mail: "Mail provider" {
  shape: cloud
}
n_queue: "Queue" {
  shape: queue
}
n_w1: "Worker 1" {
  shape: rectangle
  style.multiple: true
}
n_w2: "Worker 2" {
  shape: rectangle
  style.multiple: true
}
n_dlq: "Dead-letter queue" {
  shape: queue
}
n_campaign: "Newsletter job" {
  shape: rectangle
  style.multiple: true
}

n_buyers -> n_api: "call"
n_api -> n_queue: "async"
n_queue -> n_w1: "async"
n_w1 -> n_mail: "call"
n_queue -> n_w2: "async"
n_w2 -> n_mail: "call"
n_queue -> n_dlq: "async" {
  target-arrowhead.shape: cross
}
n_campaign -> n_queue: "async"

PlantUML: request sequence

@startuml
participant "Shop API" as n_api
boundary "Mail provider" as n_mail
queue "Queue" as n_queue
participant "Worker 2" as n_w2
participant "Worker 1" as n_w1
queue "Dead-letter queue" as n_dlq
participant "Newsletter job" as n_campaign
note over n_api, n_mail : The email is sent inside the order request
n_api -> n_mail : POST /v1/send HTTP/1.1
n_mail --> n_api : HTTP/1.1 202 Accepted
note over n_api, n_mail : A sale: 600 orders a second
n_api -> n_mail : POST /v1/send HTTP/1.1
n_mail --> n_api : HTTP/1.1 429 Too Many Requests
note over n_api, n_queue : A queue between the order and the email
n_api ->> n_queue : SEND emails
n_queue -->> n_api : OK queued, depth 42000
note over n_queue, n_w2 : More workers, up to the provider's cap
n_queue ->> n_w2 : autoscaler: queue emails
note over n_queue, n_w1 : A worker dies holding a message
n_queue ->> n_w1 : RECEIVE emails
n_w1 -->> n_queue : DELETE m_8812
note over n_queue, n_w2 : A slow job outlives its lease
n_queue ->> n_w2 : [duplicate] 10:02:00 w1 got m_9120 (receipt: 212 pages)
note over n_queue, n_w1 : Extend the lease while working
n_queue ->> n_w1 : EXTEND emails m_9120 visibility_timeout=60s
n_w1 -->> n_queue : OK until 10:03:20
note over n_w1, n_mail : The provider is down, workers hammer it
n_w1 -> n_mail : [retry] POST /v1/send HTTP/1.1
n_mail --> n_w1 : HTTP/1.1 503 Service Unavailable
note over n_queue, n_w1 : Exponential backoff with jitter
n_queue ->> n_w1 : NACK emails m_8812 delay=86s
note over n_queue, n_dlq : A poison message goes to the dead-letter queue
n_queue ->> n_dlq : SET emails redrive_policy
n_dlq -->> n_queue : m_7731 moved to emails-dlq after 5 receives
note over n_campaign, n_queue : Backpressure: the producer slows down
n_campaign ->> n_queue : campaign: 2000000 newsletter emails to publish
@enduml

PlantUML: components

@startuml
left to right direction
agent "Buyers" as n_buyers
component "Shop API" as n_api
cloud "Mail provider" as n_mail
queue "Queue" as n_queue
collections "Worker 1" as n_w1
collections "Worker 2" as n_w2
queue "Dead-letter queue" as n_dlq
collections "Newsletter job" as n_campaign
n_buyers --> n_api : call
n_api --> n_queue : async
n_queue --> n_w1 : async
n_w1 --> n_mail : call
n_queue --> n_w2 : async
n_w2 --> n_mail : call
n_queue -[#c62828]-> n_dlq : async
n_campaign --> n_queue : async
@enduml

Recap

Six rules for a queue.

1) Slow and limited work leaves the request path and goes through a queue. 2) Scale workers by backlog and the oldest message's age, up to the downstream limit. 3) Ack after the work; extend the lease for long jobs. 4) Retry with exponential backoff and jitter. 5) Poison messages go to a watched dead-letter queue. 6) Bound the queue and let producers slow down.

Numbers: at the start and at the end

MetricStartEnd
Order p95950 ms35 ms
Backlog020 000
Email delay1 s60 s
Lost emails0/h0/h

Steps

  1. Case The email is sent inside the order request

    When an order is paid, the API calls the mail provider and waits for its answer before replying to the buyer.

  2. Problem A sale: 600 orders a second

    At 10:00 a sale starts and orders jump fifteen times.

  3. Fix A queue between the order and the email

    The API now only puts a small job in a queue and answers the buyer at once.

  4. Fix More workers, up to the provider's cap

    The autoscaler watches the backlog and the age of the oldest message, and starts a second worker.

  5. Case A worker dies holding a message

    Worker 2 takes message m_8812 and dies before sending it.

  6. Problem A slow job outlives its lease

    A wholesale order has a 212-page receipt, and the PDF takes 45 seconds.

  7. Fix Extend the lease while working

    While a job runs, the worker extends its lease every 20 seconds.

  8. Problem The provider is down, workers hammer it

    The mail provider goes down for twenty minutes.

  9. Fix Exponential backoff with jitter

    After a failure the worker returns the message with a delay that doubles each time: 10 s, 20 s, 40 s, 80 s, capped at 15 minutes.

  10. Fix A poison message goes to the dead-letter queue

    One message refers to an email template that was deleted.

  11. Fix Backpressure: the producer slows down

    Marketing launches a newsletter: 2 million emails pushed into the same queue at once.

  12. Recap Recap

    Six rules for a queue.

Terms

Message queue
A buffer between a producer and workers: slow or limited work leaves the request path and goes into the queue, and workers process it at their own pace. The queue absorbs spikes; ack comes after the work.
Dead-letter queue
A separate queue that receives a poison message after it fails several attempts. It stops blocking the main queue and is watched by people.
Visibility timeout (lease)
When a queue hands a message to a worker it does not delete it, it hides it from others for this long. If the worker does not ack in time, the message becomes visible again and goes to another worker.