Cheat sheet · Communication
Sending webhooks reliably
The problem
A payment service tells online shops that a payment went through by calling their URL: a webhook. How to send it without blocking payments, how the shop knows the call is real and fresh, and how both sides survive retries, duplicates, out-of-order events and endpoints that are gone.
What you learn
- Why webhooks go through a queue and a sender, not inside the payment request
- HMAC signatures over id, timestamp and body, and replay protection
- Retries with backoff, and an idempotent receiver that answers fast
- Ordering by version, and handling endpoints that stay dead for days
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_payments["Payments API"]
n_shop>"Shop server"]
n_shopdb[("Shop DB")]
n_queue[["Webhook queue"]]
n_sender("Sender")
n_oldshop>"Dead endpoint"]
n_shop -->|"call"| n_shopdb
n_payments -->|"async"| n_queue
n_queue -->|"async"| n_sender
n_sender -->|"call"| n_shop
n_sender --x n_oldshop
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_oldshop downMermaid: request sequence
sequenceDiagram
participant n_payments as Payments API
participant n_shop as Shop server
participant n_sender as Sender
actor n_attacker as Attacker
participant n_shopdb as Shop DB
participant n_oldshop as Dead endpoint
Note over n_payments,n_shop: The webhook is sent inside the payment
n_payments->>n_shop: POST /hooks/payments HTTP/1.1
n_shop-->>n_payments: HTTP/1.1 200 OK
Note over n_sender,n_shop: A queue and a sender take the webhook out of the payment
n_sender->>n_shop: sender: deliver evt_9f2 to https://shop.example.com/hooks/payments
n_shop-->>n_sender: HTTP/1.1 200 OK
Note over n_attacker,n_shop: Anyone can call the shop's webhook URL
n_attacker->>n_shop: POST /hooks/payments HTTP/1.1
n_shop-->>n_attacker: HTTP/1.1 200 OK
Note over n_sender,n_shop: Sign every webhook with HMAC
n_sender->>n_shop: POST /hooks/payments HTTP/1.1
Note over n_attacker,n_shop: A real request, sent again
n_attacker->>n_shop: POST /hooks/payments (copied from a leaked proxy log)
Note over n_attacker,n_shop: Signed timestamp, a time window and a seen-id list
n_attacker-xn_shop: shop: verify evt_9f2
Note over n_sender,n_shop: The shop is down: retry on a schedule
n_sender->>n_shop: POST /hooks/payments HTTP/1.1
n_shop-->>n_sender: HTTP/1.1 503 Service Unavailable
Note over n_sender,n_shop: A slow answer turns into a double bonus
n_sender->>n_shop: 10:00:00 deliver evt_9f2 attempt 1
Note over n_shop,n_shopdb: Receiver: remember the id, answer fast, work later
n_shop->>n_shopdb: INSERT INTO webhook_inbox (webhook_id, received_at)
n_shopdb-->>n_shop: HTTP/1.1 200 OK
Note over n_sender,n_shop: Events arrive out of order
n_sender->>n_shop: POST /hooks/payments HTTP/1.1
n_shop-->>n_sender: shop: payment pay_771 has version 2, applying 3
Note over n_sender,n_oldshop: An endpoint that never comes back
n_sender->>n_oldshop: endpoint ep_311: failing for 5 daysD2
direction: right
n_payments: "Payments API" {
shape: rectangle
}
n_shop: "Shop server" {
shape: cloud
}
n_shopdb: "Shop DB" {
shape: cylinder
}
n_queue: "Webhook queue" {
shape: queue
}
n_sender: "Sender" {
shape: rectangle
style.multiple: true
}
n_oldshop: "Dead endpoint" {
shape: cloud
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_shop -> n_shopdb: "call"
n_payments -> n_queue: "async"
n_queue -> n_sender: "async"
n_sender -> n_shop: "call"
n_sender -> n_oldshop {
target-arrowhead.shape: cross
}PlantUML: request sequence
@startuml
participant "Payments API" as n_payments
boundary "Shop server" as n_shop
participant "Sender" as n_sender
actor "Attacker" as n_attacker
database "Shop DB" as n_shopdb
boundary "Dead endpoint" as n_oldshop
note over n_payments, n_shop : The webhook is sent inside the payment
n_payments -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_payments : HTTP/1.1 200 OK
note over n_sender, n_shop : A queue and a sender take the webhook out of the payment
n_sender -> n_shop : sender: deliver evt_9f2 to https://shop.example.com/hooks/payments
n_shop --> n_sender : HTTP/1.1 200 OK
note over n_attacker, n_shop : Anyone can call the shop's webhook URL
n_attacker -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_attacker : HTTP/1.1 200 OK
note over n_sender, n_shop : Sign every webhook with HMAC
n_sender -> n_shop : POST /hooks/payments HTTP/1.1
note over n_attacker, n_shop : A real request, sent again
n_attacker -> n_shop : [duplicate] POST /hooks/payments (copied from a leaked proxy log)
note over n_attacker, n_shop : Signed timestamp, a time window and a seen-id list
n_attacker ->x n_shop : [lost] shop: verify evt_9f2
note over n_sender, n_shop : The shop is down: retry on a schedule
n_sender -> n_shop : [retry] POST /hooks/payments HTTP/1.1
n_shop --> n_sender : HTTP/1.1 503 Service Unavailable
note over n_sender, n_shop : A slow answer turns into a double bonus
n_sender -> n_shop : [duplicate] 10:00:00 deliver evt_9f2 attempt 1
note over n_shop, n_shopdb : Receiver: remember the id, answer fast, work later
n_shop -> n_shopdb : INSERT INTO webhook_inbox (webhook_id, received_at)
n_shopdb --> n_shop : HTTP/1.1 200 OK
note over n_sender, n_shop : Events arrive out of order
n_sender -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_sender : shop: payment pay_771 has version 2, applying 3
note over n_sender, n_oldshop : An endpoint that never comes back
n_sender -> n_oldshop : endpoint ep_311: failing for 5 days
@endumlPlantUML: components
@startuml
left to right direction
component "Payments API" as n_payments
cloud "Shop server" as n_shop
database "Shop DB" as n_shopdb
queue "Webhook queue" as n_queue
collections "Sender" as n_sender
cloud "Dead endpoint" as n_oldshop #fde2e2;line:c62828;line.dashed
n_shop --> n_shopdb : call
n_payments --> n_queue : async
n_queue --> n_sender : async
n_sender --> n_shop : call
n_sender -[#c62828]-> n_oldshop
@endumlRecap
Six rules for webhooks.
1) Send from a queue with a timeout, never inside the business request. 2) Sign id, timestamp and raw body with a per-endpoint secret; verify in constant time. 3) Reject old timestamps and remember seen ids. 4) Retry on a growing schedule with jitter. 5) The receiver stores the id with a unique key, answers fast and works in the background. 6) Do not rely on order: use versions or fetch current state; isolate and disable dead endpoints.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Payment p95 | 8 200 ms | 60 ms |
| Fake events accepted | 0 | 0 |
| Double fulfilments | 0/day | 0/day |
| Pending deliveries | 0 | 9 000 |
Steps
- Case The webhook is sent inside the payment
When a card payment succeeds, the Payments API calls the shop's URL with the news and waits for the answer before replying to the buyer.
- Fix A queue and a sender take the webhook out of the payment
The Payments API now stores the event and answers the buyer at once.
- Problem Anyone can call the shop's webhook URL
The URL is not a secret: it shows up in browser tools, logs and support tickets.
- Fix Sign every webhook with HMAC
Each shop gets its own secret when it registers the endpoint.
- Problem A real request, sent again
A misconfigured proxy logged full requests, and the log leaked.
- Fix Signed timestamp, a time window and a seen-id list
The signed string is
id.timestamp.body, so the timestamp cannot be changed without breaking the signature. - Fix The shop is down: retry on a schedule
The shop's server is down for an hour during a deploy.
- Problem A slow answer turns into a double bonus
The shop handles the event inside the request: it marks the order paid and adds 500 bonus points.
- Fix Receiver: remember the id, answer fast, work later
The shop first inserts the
Webhook-Idinto an inbox table with a unique key. - Case Events arrive out of order
A payment succeeded and was refunded five minutes later.
- Fix An endpoint that never comes back
One shop closed and its domain expired.
- Recap Recap
Six rules for webhooks.
Terms
- Webhook
- When an event happens, one system sends an HTTP POST to another system's URL on its own. Delivery is at least once: the sender retries with backoff and the receiver drops repeats by event id.
- HMAC signature
- The message body and a timestamp are hashed with a shared secret and sent in a header. The receiver computes the same and compares: a mismatch or an old timestamp means the request is rejected.