Cheat sheet · Communication
Reliable callbacks
The problem
Another system sends you a message. What it takes so the message is never lost, never handled twice and never slows you down.
What you learn
- Why
200 OKcomes only after the message is saved - How a queue and a worker split the reply from the heavy work
- When to acknowledge (
ack): before the work or after it - Redelivery and the idempotency key
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_provider>"Sender"]
n_api["API"]
n_store[("Database")]
n_queue[["Queue"]]
n_worker("Worker")
n_provider -->|"call"| n_api
n_api -->|"call"| n_store
n_api -->|"async"| n_queue
n_queue -->|"async"| n_worker
n_worker -->|"call"| n_storeMermaid: request sequence
sequenceDiagram
participant n_provider as Sender
participant n_api as API
participant n_store as Database
participant n_queue as Queue
participant n_worker as Worker
Note over n_provider,n_api: All the work before the reply
n_provider->>n_api: POST /callbacks HTTP/1.1
n_api-->>n_provider: HTTP/1.1 200 OK
Note over n_api,n_store: 200 OK even on failure
n_api-xn_store: BEGIN#59;
Note over n_api,n_store: Save first, reply second
n_api->>n_store: INSERT INTO inbox (event_id, payload, received_at)
n_store-->>n_api: INSERT 0 1
Note over n_api,n_queue: Heavy work goes to a queue
n_api->>n_queue: PUBLISH callbacks.work
Note over n_queue,n_worker: Acknowledged too early
n_queue-xn_worker: DELIVER callbacks.work delivery_tag=7
Note over n_queue,n_worker: Ack only when the work is done
n_queue->>n_worker: DELIVER callbacks.work delivery_tag=7 redelivered=true
n_worker-->>n_queue: ACK delivery_tag=7
Note over n_provider,n_api: One message, two results
n_provider->>n_api: POST /callbacks HTTP/1.1
Note over n_worker,n_store: The idempotency key
n_worker->>n_store: INSERT INTO processed (event_id, result)
n_store-->>n_worker: INSERT 0 0D2
direction: right
n_provider: "Sender" {
shape: cloud
}
n_api: "API" {
shape: rectangle
}
n_store: "Database" {
shape: cylinder
}
n_queue: "Queue" {
shape: queue
}
n_worker: "Worker" {
shape: rectangle
style.multiple: true
}
n_provider -> n_api: "call"
n_api -> n_store: "call"
n_api -> n_queue: "async"
n_queue -> n_worker: "async"
n_worker -> n_store: "call"PlantUML: request sequence
@startuml
boundary "Sender" as n_provider
participant "API" as n_api
database "Database" as n_store
queue "Queue" as n_queue
participant "Worker" as n_worker
note over n_provider, n_api : All the work before the reply
n_provider -> n_api : POST /callbacks HTTP/1.1
n_api --> n_provider : HTTP/1.1 200 OK
note over n_api, n_store : 200 OK even on failure
n_api ->x n_store : [lost] BEGIN;
note over n_api, n_store : Save first, reply second
n_api -> n_store : INSERT INTO inbox (event_id, payload, received_at)
n_store --> n_api : INSERT 0 1
note over n_api, n_queue : Heavy work goes to a queue
n_api ->> n_queue : PUBLISH callbacks.work
note over n_queue, n_worker : Acknowledged too early
n_queue ->x n_worker : [lost] DELIVER callbacks.work delivery_tag=7
note over n_queue, n_worker : Ack only when the work is done
n_queue ->> n_worker : [retry] DELIVER callbacks.work delivery_tag=7 redelivered=true
n_worker -->> n_queue : ACK delivery_tag=7
note over n_provider, n_api : One message, two results
n_provider -> n_api : [duplicate] POST /callbacks HTTP/1.1
note over n_worker, n_store : The idempotency key
n_worker -> n_store : INSERT INTO processed (event_id, result)
n_store --> n_worker : INSERT 0 0
@endumlPlantUML: components
@startuml
left to right direction
cloud "Sender" as n_provider
component "API" as n_api
database "Database" as n_store
queue "Queue" as n_queue
collections "Worker" as n_worker
n_provider --> n_api : call
n_api --> n_store : call
n_api --> n_queue : async
n_queue --> n_worker : async
n_worker --> n_store : call
@endumlRecap
Four rules for a reliable callback.
1) 200 only after the message is saved. 2) Heavy work in a queue and a worker. 3) Ack after the work. 4) Every handler is idempotent: a repeat gives the same result.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Incoming | 40/s | 40/s |
| Reply p95 | 3 800 ms | 30 ms |
| Lost | 0% | 0% |
| Handled twice | 0/min | 0/min |
Steps
- Problem All the work before the reply
The sender posts a callback.
- Problem
200 OKeven on failureThe database write failed half way.
- Fix Save first, reply second
The API now does one thing: it writes the message as received into an inbox table and replies
200at once. - Fix Heavy work goes to a queue
Once the message is saved, the API puts a small job on a queue.
- Problem Acknowledged too early
The worker tells the queue “got it” (
ack) the moment a message arrives, then starts working. - Fix Ack only when the work is done
The worker acknowledges after the result is written.
- Case One message, two results
Duplicates come from both ends: the sender retries after a timeout, and the queue redelivers after a worker crash.
- Fix The idempotency key
Every message has a stable key, such as the document ID and its status.
- Recap Recap
Four rules for a reliable callback.
Terms
- Idempotency
- Running an operation once or several times gives the same result. Messages can be delivered again, so every handler must be idempotent.