Cheat sheet · Business processes
Notification delivery
The problem
A business app texts login codes, "invoice received" notices and payment receipts. Sending them inside the request ties the app to a slow provider. An outbox, a queue and a sender worker take the provider out of the request; then come rate limits, delivery receipts, a fallback chain with deadlines, dedup, user preferences, cost and a second provider.
What you learn
- Why a notification is written to an outbox with the business event and sent by a worker, not inside the request
- Rate limits per provider with a token bucket,
429andRetry-After, and a separate lane for login codes - Accepted is not delivered: delivery receipts, a fallback chain push, SMS, email, and deadlines that stop an expired code
- One event, one message per channel: idempotency keys, quiet hours, preferences, cost per message and provider failover
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_api["Business API"]
n_sms>"SMS provider A"]
n_db[("Outbox DB")]
n_queue[["Queue"]]
n_worker("Sender worker")
n_limiter{{"Rate limiter"}}
n_sms2>"SMS provider B"]
n_api --> n_db
n_db --> n_queue
n_queue --> n_worker
n_worker --x|"lost"| n_sms
n_worker --> n_limiter
n_sms -.-> n_api
n_worker --> n_db
n_worker --> n_sms2
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_sms downMermaid: request sequence
sequenceDiagram
participant n_api as Business API
participant n_sms as SMS provider A
participant n_db as Outbox DB
participant n_queue as Queue
participant n_worker as Sender worker
participant n_limiter as Rate limiter
participant n_push as Push service
participant n_sms2 as SMS provider B
Note over n_api,n_sms: The API sends the SMS inside the request
n_api->>n_sms: POST /v1/messages HTTP/1.1
n_sms-->>n_api: HTTP/1.1 200 OK
Note over n_api,n_sms: A slow provider stops logins and everything else
n_api->>n_sms: 09:00:02 POST /auth/otp u_42: calling SMS provider
Note over n_api,n_db: Write the message to an outbox in the same transaction
n_api->>n_db: BEGIN#59;
n_db-->>n_api: COMMIT 12 ms
Note over n_queue,n_worker: A queue and a sender worker do the sending
n_queue->>n_worker: notifications.sms partition 3 offset 88120
Note over n_worker,n_sms: Month end: the provider answers 429
n_worker->>n_sms: POST /v1/messages HTTP/1.1
n_sms-->>n_worker: HTTP/1.1 429 Too Many Requests
Note over n_worker,n_limiter: A token bucket per provider, a separate lane for codes
n_worker->>n_limiter: EVAL token_bucket 1 bucket:sms_a rate=50 burst=50 take=1
n_limiter-->>n_worker: (integer) 0
Note over n_sms,n_api: Accepted is not delivered: delivery receipts
n_sms->>n_api: POST /webhooks/sms-dlr HTTP/1.1
n_api-->>n_sms: HTTP/1.1 200 OK
Note over n_worker,n_push: A fallback chain with a deadline per kind of message
n_worker->>n_push: ob_7730 invoice.received chain: push -#62; sms -#62; email
Note over n_worker,n_db: One event, one message per channel
n_worker->>n_db: INSERT INTO deliveries (event_id, channel, provider_ref)
n_db-->>n_worker: INSERT 0 0
Note over n_worker,n_db: User preferences and quiet hours
n_worker->>n_db: SELECT channels, quiet_hours, tz FROM prefs WHERE user_id = 'u_77'
n_db-->>n_worker: decision ob_7731 payment.receipt at 23:40 local
Note over n_worker,n_sms2: Cost per message and a second provider
n_worker->>n_sms2: cost ob_5590 otp_login provider=sms_bD2
direction: right
n_api: "Business API" {
shape: rectangle
}
n_sms: "SMS provider A" {
shape: cloud
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_db: "Outbox DB" {
shape: cylinder
}
n_queue: "Queue" {
shape: queue
}
n_worker: "Sender worker" {
shape: rectangle
style.multiple: true
}
n_limiter: "Rate limiter" {
shape: hexagon
}
n_sms2: "SMS provider B" {
shape: cloud
}
n_api -> n_db
n_db -> n_queue
n_queue -> n_worker
n_worker -> n_sms: "lost" {
target-arrowhead.shape: cross
}
n_worker -> n_limiter
n_sms -> n_api {
style.stroke-dash: 3
}
n_worker -> n_db
n_worker -> n_sms2PlantUML: request sequence
@startuml
participant "Business API" as n_api
boundary "SMS provider A" as n_sms
database "Outbox DB" as n_db
queue "Queue" as n_queue
participant "Sender worker" as n_worker
collections "Rate limiter" as n_limiter
boundary "Push service" as n_push
boundary "SMS provider B" as n_sms2
note over n_api, n_sms : The API sends the SMS inside the request
n_api -> n_sms : POST /v1/messages HTTP/1.1
n_sms --> n_api : HTTP/1.1 200 OK
note over n_api, n_sms : A slow provider stops logins and everything else
n_api -> n_sms : 09:00:02 POST /auth/otp u_42: calling SMS provider
note over n_api, n_db : Write the message to an outbox in the same transaction
n_api -> n_db : BEGIN;
n_db --> n_api : COMMIT 12 ms
note over n_queue, n_worker : A queue and a sender worker do the sending
n_queue -> n_worker : notifications.sms partition 3 offset 88120
note over n_worker, n_sms : Month end: the provider answers 429
n_worker -> n_sms : POST /v1/messages HTTP/1.1
n_sms --> n_worker : HTTP/1.1 429 Too Many Requests
note over n_worker, n_limiter : A token bucket per provider, a separate lane for codes
n_worker -> n_limiter : EVAL token_bucket 1 bucket:sms_a rate=50 burst=50 take=1
n_limiter --> n_worker : (integer) 0
note over n_sms, n_api : Accepted is not delivered: delivery receipts
n_sms -> n_api : POST /webhooks/sms-dlr HTTP/1.1
n_api --> n_sms : HTTP/1.1 200 OK
note over n_worker, n_push : A fallback chain with a deadline per kind of message
n_worker -> n_push : ob_7730 invoice.received chain: push -> sms -> email
note over n_worker, n_db : One event, one message per channel
n_worker -> n_db : INSERT INTO deliveries (event_id, channel, provider_ref)
n_db --> n_worker : INSERT 0 0
note over n_worker, n_db : User preferences and quiet hours
n_worker -> n_db : SELECT channels, quiet_hours, tz FROM prefs WHERE user_id = 'u_77'
n_db --> n_worker : decision ob_7731 payment.receipt at 23:40 local
note over n_worker, n_sms2 : Cost per message and a second provider
n_worker -> n_sms2 : cost ob_5590 otp_login provider=sms_b
@endumlPlantUML: components
@startuml
left to right direction
component "Business API" as n_api
cloud "SMS provider A" as n_sms #fde2e2;line:c62828;line.dashed
database "Outbox DB" as n_db
queue "Queue" as n_queue
collections "Sender worker" as n_worker
stack "Rate limiter" as n_limiter
cloud "SMS provider B" as n_sms2
n_api --> n_db
n_db --> n_queue
n_queue --> n_worker
n_worker -[#c62828,dashed]-> n_sms : lost
n_worker --> n_limiter
n_sms ..> n_api
n_worker --> n_db
n_worker --> n_sms2
@endumlRecap
Six rules of notification delivery.
1) Never call a provider inside the user's request: write the message to an outbox in the same transaction as the event, and let a worker send it from a queue. 2) Pace sending with a token bucket per provider, honour 429 and Retry-After, and give login codes their own lane. 3) Accepted is not delivered: track receipts, and treat no receipt as unknown. 4) Use a fallback chain per kind of message with a deadline, so an expired code is never sent. 5) Dedup on event plus channel, and respect preferences and quiet hours. 6) Log the cost of every message and keep a second provider warm behind a circuit breaker.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| API p95 latency | 320 ms | 140 ms |
| Delivered | 97% | 98% |
| Duplicate messages | 0/day | 0/day |
| Messaging spend per day | 410 $ | 345 $ |
Steps
- Case The API sends the SMS inside the request
A user logs in.
- Problem A slow provider stops logins and everything else
At 09:00 the provider answers in 10 s instead of 0.3 s.
- Fix Write the message to an outbox in the same transaction
The login request now saves the code and one
outboxrow in a single transaction: which event, which channel, which template, and by when it must arrive. - Fix A queue and a sender worker do the sending
A relay reads new outbox rows and publishes them to a queue, one topic per channel.
- Problem Month end: the provider answers 429
At 18:00 on the last day of the month, 40 000 invoice notices land in the queue.
- Fix A token bucket per provider, a separate lane for codes
Before each send the worker takes a token from a shared rate limiter: one bucket per provider, 50 tokens a second, shared by all workers.
- Case Accepted is not delivered: delivery receipts
The provider's
200only means it took the message. - Fix A fallback chain with a deadline per kind of message
Each kind of message has a chain.
- Fix One event, one message per channel
The queue delivers at least once.
- Fix User preferences and quiet hours
Before sending, the worker reads the user's preferences: which channels per kind of message, quiet hours and time zone.
- Fix Cost per message and a second provider
Every send writes a cost line: provider, segments, encoding, price.
- Recap Recap
Six rules of notification delivery.
Terms
- Delivery receipt (DLR)
- A later callback from an SMS or push provider saying whether the message reached the phone. Accepted is not delivered.
- Channel fallback chain
- An ordered list of channels (push, SMS, email) per kind of message, stopping at the first confirmed delivery or at the message's deadline.
- Quiet hours
- A window in the user's local time when non-urgent notifications are held until morning. Codes and security alerts are exempt.