mufassall

Cheat sheet · Business processes

Notification delivery

14 min · Intermediate · mufassall.dtwin.uz/en/l/notification-delivery/

← Open the lesson

The problem

A business app texts login codes, "invoice received" notices and payment receipts. Sending them inside the request ties the app to a slow provider. An outbox, a queue and a sender worker take the provider out of the request; then come rate limits, delivery receipts, a fallback chain with deadlines, dedup, user preferences, cost and a second provider.

What you learn

  • Why a notification is written to an outbox with the business event and sent by a worker, not inside the request
  • Rate limits per provider with a token bucket, 429 and Retry-After, and a separate lane for login codes
  • Accepted is not delivered: delivery receipts, a fallback chain push, SMS, email, and deadlines that stop an expired code
  • One event, one message per channel: idempotency keys, quiet hours, preferences, cost per message and provider failover
Final scene
35%55%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_api["Business API"]
  n_sms>"SMS provider A"]
  n_db[("Outbox DB")]
  n_queue[["Queue"]]
  n_worker("Sender worker")
  n_limiter{{"Rate limiter"}}
  n_sms2>"SMS provider B"]
  n_api --> n_db
  n_db --> n_queue
  n_queue --> n_worker
  n_worker --x|"lost"| n_sms
  n_worker --> n_limiter
  n_sms -.-> n_api
  n_worker --> n_db
  n_worker --> n_sms2
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_sms down

Mermaid: request sequence

sequenceDiagram
  participant n_api as Business API
  participant n_sms as SMS provider A
  participant n_db as Outbox DB
  participant n_queue as Queue
  participant n_worker as Sender worker
  participant n_limiter as Rate limiter
  participant n_push as Push service
  participant n_sms2 as SMS provider B
  Note over n_api,n_sms: The API sends the SMS inside the request
  n_api->>n_sms: POST /v1/messages HTTP/1.1
  n_sms-->>n_api: HTTP/1.1 200 OK
  Note over n_api,n_sms: A slow provider stops logins and everything else
  n_api->>n_sms: 09:00:02 POST /auth/otp u_42: calling SMS provider
  Note over n_api,n_db: Write the message to an outbox in the same transaction
  n_api->>n_db: BEGIN#59;
  n_db-->>n_api: COMMIT 12 ms
  Note over n_queue,n_worker: A queue and a sender worker do the sending
  n_queue->>n_worker: notifications.sms partition 3 offset 88120
  Note over n_worker,n_sms: Month end: the provider answers 429
  n_worker->>n_sms: POST /v1/messages HTTP/1.1
  n_sms-->>n_worker: HTTP/1.1 429 Too Many Requests
  Note over n_worker,n_limiter: A token bucket per provider, a separate lane for codes
  n_worker->>n_limiter: EVAL token_bucket 1 bucket:sms_a rate=50 burst=50 take=1
  n_limiter-->>n_worker: (integer) 0
  Note over n_sms,n_api: Accepted is not delivered: delivery receipts
  n_sms->>n_api: POST /webhooks/sms-dlr HTTP/1.1
  n_api-->>n_sms: HTTP/1.1 200 OK
  Note over n_worker,n_push: A fallback chain with a deadline per kind of message
  n_worker->>n_push: ob_7730 invoice.received chain: push -#62; sms -#62; email
  Note over n_worker,n_db: One event, one message per channel
  n_worker->>n_db: INSERT INTO deliveries (event_id, channel, provider_ref)
  n_db-->>n_worker: INSERT 0 0
  Note over n_worker,n_db: User preferences and quiet hours
  n_worker->>n_db: SELECT channels, quiet_hours, tz FROM prefs WHERE user_id = 'u_77'
  n_db-->>n_worker: decision ob_7731 payment.receipt at 23:40 local
  Note over n_worker,n_sms2: Cost per message and a second provider
  n_worker->>n_sms2: cost ob_5590 otp_login provider=sms_b

D2

direction: right

n_api: "Business API" {
  shape: rectangle
}
n_sms: "SMS provider A" {
  shape: cloud
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_db: "Outbox DB" {
  shape: cylinder
}
n_queue: "Queue" {
  shape: queue
}
n_worker: "Sender worker" {
  shape: rectangle
  style.multiple: true
}
n_limiter: "Rate limiter" {
  shape: hexagon
}
n_sms2: "SMS provider B" {
  shape: cloud
}

n_api -> n_db
n_db -> n_queue
n_queue -> n_worker
n_worker -> n_sms: "lost" {
  target-arrowhead.shape: cross
}
n_worker -> n_limiter
n_sms -> n_api {
  style.stroke-dash: 3
}
n_worker -> n_db
n_worker -> n_sms2

PlantUML: request sequence

@startuml
participant "Business API" as n_api
boundary "SMS provider A" as n_sms
database "Outbox DB" as n_db
queue "Queue" as n_queue
participant "Sender worker" as n_worker
collections "Rate limiter" as n_limiter
boundary "Push service" as n_push
boundary "SMS provider B" as n_sms2
note over n_api, n_sms : The API sends the SMS inside the request
n_api -> n_sms : POST /v1/messages HTTP/1.1
n_sms --> n_api : HTTP/1.1 200 OK
note over n_api, n_sms : A slow provider stops logins and everything else
n_api -> n_sms : 09:00:02 POST /auth/otp u_42: calling SMS provider
note over n_api, n_db : Write the message to an outbox in the same transaction
n_api -> n_db : BEGIN;
n_db --> n_api : COMMIT 12 ms
note over n_queue, n_worker : A queue and a sender worker do the sending
n_queue -> n_worker : notifications.sms partition 3 offset 88120
note over n_worker, n_sms : Month end: the provider answers 429
n_worker -> n_sms : POST /v1/messages HTTP/1.1
n_sms --> n_worker : HTTP/1.1 429 Too Many Requests
note over n_worker, n_limiter : A token bucket per provider, a separate lane for codes
n_worker -> n_limiter : EVAL token_bucket 1 bucket:sms_a rate=50 burst=50 take=1
n_limiter --> n_worker : (integer) 0
note over n_sms, n_api : Accepted is not delivered: delivery receipts
n_sms -> n_api : POST /webhooks/sms-dlr HTTP/1.1
n_api --> n_sms : HTTP/1.1 200 OK
note over n_worker, n_push : A fallback chain with a deadline per kind of message
n_worker -> n_push : ob_7730 invoice.received chain: push -> sms -> email
note over n_worker, n_db : One event, one message per channel
n_worker -> n_db : INSERT INTO deliveries (event_id, channel, provider_ref)
n_db --> n_worker : INSERT 0 0
note over n_worker, n_db : User preferences and quiet hours
n_worker -> n_db : SELECT channels, quiet_hours, tz FROM prefs WHERE user_id = 'u_77'
n_db --> n_worker : decision ob_7731 payment.receipt at 23:40 local
note over n_worker, n_sms2 : Cost per message and a second provider
n_worker -> n_sms2 : cost ob_5590 otp_login provider=sms_b
@enduml

PlantUML: components

@startuml
left to right direction
component "Business API" as n_api
cloud "SMS provider A" as n_sms #fde2e2;line:c62828;line.dashed
database "Outbox DB" as n_db
queue "Queue" as n_queue
collections "Sender worker" as n_worker
stack "Rate limiter" as n_limiter
cloud "SMS provider B" as n_sms2
n_api --> n_db
n_db --> n_queue
n_queue --> n_worker
n_worker -[#c62828,dashed]-> n_sms : lost
n_worker --> n_limiter
n_sms ..> n_api
n_worker --> n_db
n_worker --> n_sms2
@enduml

Recap

Six rules of notification delivery.

1) Never call a provider inside the user's request: write the message to an outbox in the same transaction as the event, and let a worker send it from a queue. 2) Pace sending with a token bucket per provider, honour 429 and Retry-After, and give login codes their own lane. 3) Accepted is not delivered: track receipts, and treat no receipt as unknown. 4) Use a fallback chain per kind of message with a deadline, so an expired code is never sent. 5) Dedup on event plus channel, and respect preferences and quiet hours. 6) Log the cost of every message and keep a second provider warm behind a circuit breaker.

Numbers: at the start and at the end

MetricStartEnd
API p95 latency320 ms140 ms
Delivered97%98%
Duplicate messages0/day0/day
Messaging spend per day410 $345 $

Steps

  1. Case The API sends the SMS inside the request

    A user logs in.

  2. Problem A slow provider stops logins and everything else

    At 09:00 the provider answers in 10 s instead of 0.3 s.

  3. Fix Write the message to an outbox in the same transaction

    The login request now saves the code and one outbox row in a single transaction: which event, which channel, which template, and by when it must arrive.

  4. Fix A queue and a sender worker do the sending

    A relay reads new outbox rows and publishes them to a queue, one topic per channel.

  5. Problem Month end: the provider answers 429

    At 18:00 on the last day of the month, 40 000 invoice notices land in the queue.

  6. Fix A token bucket per provider, a separate lane for codes

    Before each send the worker takes a token from a shared rate limiter: one bucket per provider, 50 tokens a second, shared by all workers.

  7. Case Accepted is not delivered: delivery receipts

    The provider's 200 only means it took the message.

  8. Fix A fallback chain with a deadline per kind of message

    Each kind of message has a chain.

  9. Fix One event, one message per channel

    The queue delivers at least once.

  10. Fix User preferences and quiet hours

    Before sending, the worker reads the user's preferences: which channels per kind of message, quiet hours and time zone.

  11. Fix Cost per message and a second provider

    Every send writes a cost line: provider, segments, encoding, price.

  12. Recap Recap

    Six rules of notification delivery.

Terms

Delivery receipt (DLR)
A later callback from an SMS or push provider saying whether the message reached the phone. Accepted is not delivered.
Channel fallback chain
An ordered list of channels (push, SMS, email) per kind of message, stopping at the first confirmed delivery or at the message's deadline.
Quiet hours
A window in the user's local time when non-urgent notifications are held until morning. Codes and security alerts are exempt.