mufassall

Cheat sheet · Communication

Reliable callbacks

12 min · Intermediate · mufassall.dtwin.uz/en/l/callbacks/

← Open the lesson

The problem

Another system sends you a message. What it takes so the message is never lost, never handled twice and never slows you down.

What you learn

  • Why 200 OK comes only after the message is saved
  • How a queue and a worker split the reply from the heavy work
  • When to acknowledge (ack): before the work or after it
  • Redelivery and the idempotency key
Final scene

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_provider>"Sender"]
  n_api["API"]
  n_store[("Database")]
  n_queue[["Queue"]]
  n_worker("Worker")
  n_provider -->|"call"| n_api
  n_api -->|"call"| n_store
  n_api -->|"async"| n_queue
  n_queue -->|"async"| n_worker
  n_worker -->|"call"| n_store

Mermaid: request sequence

sequenceDiagram
  participant n_provider as Sender
  participant n_api as API
  participant n_store as Database
  participant n_queue as Queue
  participant n_worker as Worker
  Note over n_provider,n_api: All the work before the reply
  n_provider->>n_api: POST /callbacks HTTP/1.1
  n_api-->>n_provider: HTTP/1.1 200 OK
  Note over n_api,n_store: 200 OK even on failure
  n_api-xn_store: BEGIN#59;
  Note over n_api,n_store: Save first, reply second
  n_api->>n_store: INSERT INTO inbox (event_id, payload, received_at)
  n_store-->>n_api: INSERT 0 1
  Note over n_api,n_queue: Heavy work goes to a queue
  n_api->>n_queue: PUBLISH callbacks.work
  Note over n_queue,n_worker: Acknowledged too early
  n_queue-xn_worker: DELIVER callbacks.work delivery_tag=7
  Note over n_queue,n_worker: Ack only when the work is done
  n_queue->>n_worker: DELIVER callbacks.work delivery_tag=7 redelivered=true
  n_worker-->>n_queue: ACK delivery_tag=7
  Note over n_provider,n_api: One message, two results
  n_provider->>n_api: POST /callbacks HTTP/1.1
  Note over n_worker,n_store: The idempotency key
  n_worker->>n_store: INSERT INTO processed (event_id, result)
  n_store-->>n_worker: INSERT 0 0

D2

direction: right

n_provider: "Sender" {
  shape: cloud
}
n_api: "API" {
  shape: rectangle
}
n_store: "Database" {
  shape: cylinder
}
n_queue: "Queue" {
  shape: queue
}
n_worker: "Worker" {
  shape: rectangle
  style.multiple: true
}

n_provider -> n_api: "call"
n_api -> n_store: "call"
n_api -> n_queue: "async"
n_queue -> n_worker: "async"
n_worker -> n_store: "call"

PlantUML: request sequence

@startuml
boundary "Sender" as n_provider
participant "API" as n_api
database "Database" as n_store
queue "Queue" as n_queue
participant "Worker" as n_worker
note over n_provider, n_api : All the work before the reply
n_provider -> n_api : POST /callbacks HTTP/1.1
n_api --> n_provider : HTTP/1.1 200 OK
note over n_api, n_store : 200 OK even on failure
n_api ->x n_store : [lost] BEGIN;
note over n_api, n_store : Save first, reply second
n_api -> n_store : INSERT INTO inbox (event_id, payload, received_at)
n_store --> n_api : INSERT 0 1
note over n_api, n_queue : Heavy work goes to a queue
n_api ->> n_queue : PUBLISH callbacks.work
note over n_queue, n_worker : Acknowledged too early
n_queue ->x n_worker : [lost] DELIVER callbacks.work delivery_tag=7
note over n_queue, n_worker : Ack only when the work is done
n_queue ->> n_worker : [retry] DELIVER callbacks.work delivery_tag=7 redelivered=true
n_worker -->> n_queue : ACK delivery_tag=7
note over n_provider, n_api : One message, two results
n_provider -> n_api : [duplicate] POST /callbacks HTTP/1.1
note over n_worker, n_store : The idempotency key
n_worker -> n_store : INSERT INTO processed (event_id, result)
n_store --> n_worker : INSERT 0 0
@enduml

PlantUML: components

@startuml
left to right direction
cloud "Sender" as n_provider
component "API" as n_api
database "Database" as n_store
queue "Queue" as n_queue
collections "Worker" as n_worker
n_provider --> n_api : call
n_api --> n_store : call
n_api --> n_queue : async
n_queue --> n_worker : async
n_worker --> n_store : call
@enduml

Recap

Four rules for a reliable callback.

1) 200 only after the message is saved. 2) Heavy work in a queue and a worker. 3) Ack after the work. 4) Every handler is idempotent: a repeat gives the same result.

Numbers: at the start and at the end

MetricStartEnd
Incoming40/s40/s
Reply p953 800 ms30 ms
Lost0%0%
Handled twice0/min0/min

Steps

  1. Problem All the work before the reply

    The sender posts a callback.

  2. Problem 200 OK even on failure

    The database write failed half way.

  3. Fix Save first, reply second

    The API now does one thing: it writes the message as received into an inbox table and replies 200 at once.

  4. Fix Heavy work goes to a queue

    Once the message is saved, the API puts a small job on a queue.

  5. Problem Acknowledged too early

    The worker tells the queue “got it” (ack) the moment a message arrives, then starts working.

  6. Fix Ack only when the work is done

    The worker acknowledges after the result is written.

  7. Case One message, two results

    Duplicates come from both ends: the sender retries after a timeout, and the queue redelivers after a worker crash.

  8. Fix The idempotency key

    Every message has a stable key, such as the document ID and its status.

  9. Recap Recap

    Four rules for a reliable callback.

Terms

Idempotency
Running an operation once or several times gives the same result. Messages can be delivered again, so every handler must be idempotent.