mufassall

Cheat sheet · Communication

Sending webhooks reliably

14 min · Intermediate · mufassall.dtwin.uz/en/l/webhooks/

← Open the lesson

The problem

A payment service tells online shops that a payment went through by calling their URL: a webhook. How to send it without blocking payments, how the shop knows the call is real and fresh, and how both sides survive retries, duplicates, out-of-order events and endpoints that are gone.

What you learn

  • Why webhooks go through a queue and a sender, not inside the payment request
  • HMAC signatures over id, timestamp and body, and replay protection
  • Retries with backoff, and an idempotent receiver that answers fast
  • Ordering by version, and handling endpoints that stay dead for days
Final scene
15%25%70%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_payments["Payments API"]
  n_shop>"Shop server"]
  n_shopdb[("Shop DB")]
  n_queue[["Webhook queue"]]
  n_sender("Sender")
  n_oldshop>"Dead endpoint"]
  n_shop -->|"call"| n_shopdb
  n_payments -->|"async"| n_queue
  n_queue -->|"async"| n_sender
  n_sender -->|"call"| n_shop
  n_sender --x n_oldshop
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_oldshop down

Mermaid: request sequence

sequenceDiagram
  participant n_payments as Payments API
  participant n_shop as Shop server
  participant n_sender as Sender
  actor n_attacker as Attacker
  participant n_shopdb as Shop DB
  participant n_oldshop as Dead endpoint
  Note over n_payments,n_shop: The webhook is sent inside the payment
  n_payments->>n_shop: POST /hooks/payments HTTP/1.1
  n_shop-->>n_payments: HTTP/1.1 200 OK
  Note over n_sender,n_shop: A queue and a sender take the webhook out of the payment
  n_sender->>n_shop: sender: deliver evt_9f2 to https://shop.example.com/hooks/payments
  n_shop-->>n_sender: HTTP/1.1 200 OK
  Note over n_attacker,n_shop: Anyone can call the shop's webhook URL
  n_attacker->>n_shop: POST /hooks/payments HTTP/1.1
  n_shop-->>n_attacker: HTTP/1.1 200 OK
  Note over n_sender,n_shop: Sign every webhook with HMAC
  n_sender->>n_shop: POST /hooks/payments HTTP/1.1
  Note over n_attacker,n_shop: A real request, sent again
  n_attacker->>n_shop: POST /hooks/payments (copied from a leaked proxy log)
  Note over n_attacker,n_shop: Signed timestamp, a time window and a seen-id list
  n_attacker-xn_shop: shop: verify evt_9f2
  Note over n_sender,n_shop: The shop is down: retry on a schedule
  n_sender->>n_shop: POST /hooks/payments HTTP/1.1
  n_shop-->>n_sender: HTTP/1.1 503 Service Unavailable
  Note over n_sender,n_shop: A slow answer turns into a double bonus
  n_sender->>n_shop: 10:00:00 deliver evt_9f2 attempt 1
  Note over n_shop,n_shopdb: Receiver: remember the id, answer fast, work later
  n_shop->>n_shopdb: INSERT INTO webhook_inbox (webhook_id, received_at)
  n_shopdb-->>n_shop: HTTP/1.1 200 OK
  Note over n_sender,n_shop: Events arrive out of order
  n_sender->>n_shop: POST /hooks/payments HTTP/1.1
  n_shop-->>n_sender: shop: payment pay_771 has version 2, applying 3
  Note over n_sender,n_oldshop: An endpoint that never comes back
  n_sender->>n_oldshop: endpoint ep_311: failing for 5 days

D2

direction: right

n_payments: "Payments API" {
  shape: rectangle
}
n_shop: "Shop server" {
  shape: cloud
}
n_shopdb: "Shop DB" {
  shape: cylinder
}
n_queue: "Webhook queue" {
  shape: queue
}
n_sender: "Sender" {
  shape: rectangle
  style.multiple: true
}
n_oldshop: "Dead endpoint" {
  shape: cloud
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}

n_shop -> n_shopdb: "call"
n_payments -> n_queue: "async"
n_queue -> n_sender: "async"
n_sender -> n_shop: "call"
n_sender -> n_oldshop {
  target-arrowhead.shape: cross
}

PlantUML: request sequence

@startuml
participant "Payments API" as n_payments
boundary "Shop server" as n_shop
participant "Sender" as n_sender
actor "Attacker" as n_attacker
database "Shop DB" as n_shopdb
boundary "Dead endpoint" as n_oldshop
note over n_payments, n_shop : The webhook is sent inside the payment
n_payments -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_payments : HTTP/1.1 200 OK
note over n_sender, n_shop : A queue and a sender take the webhook out of the payment
n_sender -> n_shop : sender: deliver evt_9f2 to https://shop.example.com/hooks/payments
n_shop --> n_sender : HTTP/1.1 200 OK
note over n_attacker, n_shop : Anyone can call the shop's webhook URL
n_attacker -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_attacker : HTTP/1.1 200 OK
note over n_sender, n_shop : Sign every webhook with HMAC
n_sender -> n_shop : POST /hooks/payments HTTP/1.1
note over n_attacker, n_shop : A real request, sent again
n_attacker -> n_shop : [duplicate] POST /hooks/payments (copied from a leaked proxy log)
note over n_attacker, n_shop : Signed timestamp, a time window and a seen-id list
n_attacker ->x n_shop : [lost] shop: verify evt_9f2
note over n_sender, n_shop : The shop is down: retry on a schedule
n_sender -> n_shop : [retry] POST /hooks/payments HTTP/1.1
n_shop --> n_sender : HTTP/1.1 503 Service Unavailable
note over n_sender, n_shop : A slow answer turns into a double bonus
n_sender -> n_shop : [duplicate] 10:00:00 deliver evt_9f2 attempt 1
note over n_shop, n_shopdb : Receiver: remember the id, answer fast, work later
n_shop -> n_shopdb : INSERT INTO webhook_inbox (webhook_id, received_at)
n_shopdb --> n_shop : HTTP/1.1 200 OK
note over n_sender, n_shop : Events arrive out of order
n_sender -> n_shop : POST /hooks/payments HTTP/1.1
n_shop --> n_sender : shop: payment pay_771 has version 2, applying 3
note over n_sender, n_oldshop : An endpoint that never comes back
n_sender -> n_oldshop : endpoint ep_311: failing for 5 days
@enduml

PlantUML: components

@startuml
left to right direction
component "Payments API" as n_payments
cloud "Shop server" as n_shop
database "Shop DB" as n_shopdb
queue "Webhook queue" as n_queue
collections "Sender" as n_sender
cloud "Dead endpoint" as n_oldshop #fde2e2;line:c62828;line.dashed
n_shop --> n_shopdb : call
n_payments --> n_queue : async
n_queue --> n_sender : async
n_sender --> n_shop : call
n_sender -[#c62828]-> n_oldshop
@enduml

Recap

Six rules for webhooks.

1) Send from a queue with a timeout, never inside the business request. 2) Sign id, timestamp and raw body with a per-endpoint secret; verify in constant time. 3) Reject old timestamps and remember seen ids. 4) Retry on a growing schedule with jitter. 5) The receiver stores the id with a unique key, answers fast and works in the background. 6) Do not rely on order: use versions or fetch current state; isolate and disable dead endpoints.

Numbers: at the start and at the end

MetricStartEnd
Payment p958 200 ms60 ms
Fake events accepted00
Double fulfilments0/day0/day
Pending deliveries09 000

Steps

  1. Case The webhook is sent inside the payment

    When a card payment succeeds, the Payments API calls the shop's URL with the news and waits for the answer before replying to the buyer.

  2. Fix A queue and a sender take the webhook out of the payment

    The Payments API now stores the event and answers the buyer at once.

  3. Problem Anyone can call the shop's webhook URL

    The URL is not a secret: it shows up in browser tools, logs and support tickets.

  4. Fix Sign every webhook with HMAC

    Each shop gets its own secret when it registers the endpoint.

  5. Problem A real request, sent again

    A misconfigured proxy logged full requests, and the log leaked.

  6. Fix Signed timestamp, a time window and a seen-id list

    The signed string is id.timestamp.body, so the timestamp cannot be changed without breaking the signature.

  7. Fix The shop is down: retry on a schedule

    The shop's server is down for an hour during a deploy.

  8. Problem A slow answer turns into a double bonus

    The shop handles the event inside the request: it marks the order paid and adds 500 bonus points.

  9. Fix Receiver: remember the id, answer fast, work later

    The shop first inserts the Webhook-Id into an inbox table with a unique key.

  10. Case Events arrive out of order

    A payment succeeded and was refunded five minutes later.

  11. Fix An endpoint that never comes back

    One shop closed and its domain expired.

  12. Recap Recap

    Six rules for webhooks.

Terms

Webhook
When an event happens, one system sends an HTTP POST to another system's URL on its own. Delivery is at least once: the sender retries with backoff and the receiver drops repeats by event id.
HMAC signature
The message body and a timestamp are hashed with a shared secret and sent in a header. The receiver computes the same and compares: a mismatch or an old timestamp means the request is rejected.