Cheat sheet · Architecture
Sagas and the outbox
The problem
One order needs a payment and a stock reservation in two other services, each with its own database. There is no single transaction to wrap it in. How to keep the three in agreement without locking the world.
What you learn
- Why one database transaction cannot span three services
- What two-phase commit costs and why teams avoid it
- Saga: local steps plus compensating actions, orchestrated or choreographed
- Outbox: the DB write and the event can no longer diverge
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_client(["Shop app"])
n_orders["Orders"]
n_odb[("Orders DB")]
n_payments["Payments"]
n_inventory["Inventory"]
n_bus[["Event log"]]
n_relay("Outbox relay")
n_client -->|"call"| n_orders
n_orders -->|"call"| n_odb
n_bus ==>|"duplicate"| n_payments
n_payments -.->|"async"| n_bus
n_bus -->|"async"| n_inventory
n_relay -->|"call"| n_odb
n_relay ==>|"duplicate"| n_busMermaid: request sequence
sequenceDiagram
participant n_client as Shop app
participant n_orders as Orders
participant n_inventory as Inventory
participant n_payments as Payments
participant n_odb as Orders DB
participant n_bus as Event log
Note over n_client,n_orders: One order, three databases
n_client->>n_orders: POST /orders HTTP/1.1
n_orders-->>n_client: HTTP/1.1 202 Accepted
Note over n_orders,n_inventory: Money taken, stock call failed
n_orders->>n_inventory: POST /reservations HTTP/1.1
n_inventory-->>n_orders: HTTP/1.1 503 Service Unavailable
Note over n_orders,n_payments: Two-phase commit: everyone votes first
n_orders->>n_payments: PREPARE tx=ord_1042
n_payments-->>n_orders: VOTE yes
Note over n_orders,n_payments: The coordinator dies after the vote
n_orders-xn_payments: PREPARE tx=ord_1042 -#62; VOTE yes
Note over n_orders,n_odb: Saga: a chain of local transactions
n_orders->>n_odb: INSERT INTO sagas (id, order_id, step, status)
n_odb-->>n_orders: UPDATE 1
Note over n_orders,n_payments: Failure: undo with a compensating action
n_orders->>n_payments: POST /charges/ch_88/refunds HTTP/1.1
n_payments-->>n_orders: HTTP/1.1 201 Created
Note over n_orders,n_bus: Choreography: services react to events
n_orders->>n_bus: produce topic=orders key=1042
n_bus-->>n_orders: ack partition=3 offset=7713
Note over n_orders,n_bus: Saved in the DB, event never sent
n_orders-xn_bus: produce topic=orders key=1042
Note over n_orders,n_odb: Outbox: the event is written with the order
n_orders->>n_odb: BEGIN#59;
n_odb-->>n_orders: COMMIT
Note over n_bus,n_payments: At-least-once: consumers drop duplicates
n_bus->>n_payments: deliver topic=orders key=1042 offset=7714
n_payments-->>n_bus: INSERT INTO processed (event_id) VALUES ('ev_5c1') ON CONFLICT DO NOTHINGD2
direction: right
n_client: "Shop app" {
shape: rectangle
}
n_orders: "Orders" {
shape: rectangle
}
n_odb: "Orders DB" {
shape: cylinder
}
n_payments: "Payments" {
shape: rectangle
}
n_inventory: "Inventory" {
shape: rectangle
}
n_bus: "Event log" {
shape: queue
}
n_relay: "Outbox relay" {
shape: rectangle
style.multiple: true
}
n_client -> n_orders: "call"
n_orders -> n_odb: "call"
n_bus -> n_payments: "duplicate" {
style.stroke-width: 4
}
n_payments -> n_bus: "async" {
style.stroke-dash: 3
}
n_bus -> n_inventory: "async"
n_relay -> n_odb: "call"
n_relay -> n_bus: "duplicate" {
style.stroke-width: 4
}PlantUML: request sequence
@startuml
participant "Shop app" as n_client
participant "Orders" as n_orders
participant "Inventory" as n_inventory
participant "Payments" as n_payments
database "Orders DB" as n_odb
queue "Event log" as n_bus
note over n_client, n_orders : One order, three databases
n_client -> n_orders : POST /orders HTTP/1.1
n_orders --> n_client : HTTP/1.1 202 Accepted
note over n_orders, n_inventory : Money taken, stock call failed
n_orders -> n_inventory : POST /reservations HTTP/1.1
n_inventory --> n_orders : HTTP/1.1 503 Service Unavailable
note over n_orders, n_payments : Two-phase commit: everyone votes first
n_orders -> n_payments : PREPARE tx=ord_1042
n_payments --> n_orders : VOTE yes
note over n_orders, n_payments : The coordinator dies after the vote
n_orders ->x n_payments : [lost] PREPARE tx=ord_1042 -> VOTE yes
note over n_orders, n_odb : Saga: a chain of local transactions
n_orders -> n_odb : INSERT INTO sagas (id, order_id, step, status)
n_odb --> n_orders : UPDATE 1
note over n_orders, n_payments : Failure: undo with a compensating action
n_orders -> n_payments : POST /charges/ch_88/refunds HTTP/1.1
n_payments --> n_orders : HTTP/1.1 201 Created
note over n_orders, n_bus : Choreography: services react to events
n_orders ->> n_bus : produce topic=orders key=1042
n_bus -->> n_orders : ack partition=3 offset=7713
note over n_orders, n_bus : Saved in the DB, event never sent
n_orders ->x n_bus : [lost] produce topic=orders key=1042
note over n_orders, n_odb : Outbox: the event is written with the order
n_orders -> n_odb : BEGIN;
n_odb --> n_orders : COMMIT
note over n_bus, n_payments : At-least-once: consumers drop duplicates
n_bus ->> n_payments : [duplicate] deliver topic=orders key=1042 offset=7714
n_payments -->> n_bus : INSERT INTO processed (event_id) VALUES ('ev_5c1') ON CONFLICT DO NOTHING
@endumlPlantUML: components
@startuml
left to right direction
agent "Shop app" as n_client
component "Orders" as n_orders
database "Orders DB" as n_odb
component "Payments" as n_payments
component "Inventory" as n_inventory
queue "Event log" as n_bus
collections "Outbox relay" as n_relay
n_client --> n_orders : call
n_orders --> n_odb : call
n_bus -[bold]-> n_payments : duplicate
n_payments ..> n_bus : async
n_bus --> n_inventory : async
n_relay --> n_odb : call
n_relay -[bold]-> n_bus : duplicate
@endumlRecap
Four rules for work that spans services.
1) There is no transaction across services; avoid 2PC, it blocks everyone on the slowest node. 2) Use a saga: local steps, each with a compensating action, run by an orchestrator or by events. 3) Write events through an outbox in the same transaction as the data. 4) Delivery is at least once, so every consumer dedupes by event_id.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Order p95 | 320 ms | 190 ms |
| Charged, no order | 0/day | 0/day |
| Lost events | 0/day | 0/day |
Steps
- Case One order, three databases
The shop app places an order.
- Problem Money taken, stock call failed
Payments charged the card.
- Case Two-phase commit: everyone votes first
A coordinator (here Orders) asks every participant to prepare: do the work, lock the rows, and vote.
- Problem The coordinator dies after the vote
Orders crashed between the votes and the decision.
- Fix Saga: a chain of local transactions
Drop the global lock.
- Fix Failure: undo with a compensating action
Inventory answers "out of stock".
- Case Choreography: services react to events
Another way to run the same saga: no orchestrator.
- Problem Saved in the DB, event never sent
Orders commits the order and then publishes the event: two separate writes.
- Fix Outbox: the event is written with the order
Orders writes the order and the event into an
outboxtable in one local transaction. - Fix At-least-once: consumers drop duplicates
The relay published row
ev_5c1and crashed before marking it sent. - Recap Recap
Four rules for work that spans services.
Terms
- Two-phase commit (2PC)
- A coordinator first asks every participant to prepare and vote, and sends commit only if all vote yes. Participants hold locks the whole time and get stuck if the coordinator dies, so teams avoid it.
- Saga
- Work across several services runs as a chain of local transactions, each with a compensating action. If a step fails, earlier steps are undone by their compensations; an orchestrator or events (choreography) drive the chain.
- Outbox
- A service writes the data and the event into an outbox table in one local transaction, and a separate relay publishes the event to the broker. The event can be late, but it can no longer be lost.
- At-least-once delivery
- A message is delivered at least once, and sometimes twice. So consumers drop duplicates by event_id or write idempotently.