mufassall

Cheat sheet · Architecture

Sagas and the outbox

15 min · Advanced · mufassall.dtwin.uz/en/l/sagas/

← Open the lesson

The problem

One order needs a payment and a stock reservation in two other services, each with its own database. There is no single transaction to wrap it in. How to keep the three in agreement without locking the world.

What you learn

  • Why one database transaction cannot span three services
  • What two-phase commit costs and why teams avoid it
  • Saga: local steps plus compensating actions, orchestrated or choreographed
  • Outbox: the DB write and the event can no longer diverge
Final scene
30%30%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_client(["Shop app"])
  n_orders["Orders"]
  n_odb[("Orders DB")]
  n_payments["Payments"]
  n_inventory["Inventory"]
  n_bus[["Event log"]]
  n_relay("Outbox relay")
  n_client -->|"call"| n_orders
  n_orders -->|"call"| n_odb
  n_bus ==>|"duplicate"| n_payments
  n_payments -.->|"async"| n_bus
  n_bus -->|"async"| n_inventory
  n_relay -->|"call"| n_odb
  n_relay ==>|"duplicate"| n_bus

Mermaid: request sequence

sequenceDiagram
  participant n_client as Shop app
  participant n_orders as Orders
  participant n_inventory as Inventory
  participant n_payments as Payments
  participant n_odb as Orders DB
  participant n_bus as Event log
  Note over n_client,n_orders: One order, three databases
  n_client->>n_orders: POST /orders HTTP/1.1
  n_orders-->>n_client: HTTP/1.1 202 Accepted
  Note over n_orders,n_inventory: Money taken, stock call failed
  n_orders->>n_inventory: POST /reservations HTTP/1.1
  n_inventory-->>n_orders: HTTP/1.1 503 Service Unavailable
  Note over n_orders,n_payments: Two-phase commit: everyone votes first
  n_orders->>n_payments: PREPARE tx=ord_1042
  n_payments-->>n_orders: VOTE yes
  Note over n_orders,n_payments: The coordinator dies after the vote
  n_orders-xn_payments: PREPARE tx=ord_1042 -#62; VOTE yes
  Note over n_orders,n_odb: Saga: a chain of local transactions
  n_orders->>n_odb: INSERT INTO sagas (id, order_id, step, status)
  n_odb-->>n_orders: UPDATE 1
  Note over n_orders,n_payments: Failure: undo with a compensating action
  n_orders->>n_payments: POST /charges/ch_88/refunds HTTP/1.1
  n_payments-->>n_orders: HTTP/1.1 201 Created
  Note over n_orders,n_bus: Choreography: services react to events
  n_orders->>n_bus: produce topic=orders key=1042
  n_bus-->>n_orders: ack partition=3 offset=7713
  Note over n_orders,n_bus: Saved in the DB, event never sent
  n_orders-xn_bus: produce topic=orders key=1042
  Note over n_orders,n_odb: Outbox: the event is written with the order
  n_orders->>n_odb: BEGIN#59;
  n_odb-->>n_orders: COMMIT
  Note over n_bus,n_payments: At-least-once: consumers drop duplicates
  n_bus->>n_payments: deliver topic=orders key=1042 offset=7714
  n_payments-->>n_bus: INSERT INTO processed (event_id) VALUES ('ev_5c1') ON CONFLICT DO NOTHING

D2

direction: right

n_client: "Shop app" {
  shape: rectangle
}
n_orders: "Orders" {
  shape: rectangle
}
n_odb: "Orders DB" {
  shape: cylinder
}
n_payments: "Payments" {
  shape: rectangle
}
n_inventory: "Inventory" {
  shape: rectangle
}
n_bus: "Event log" {
  shape: queue
}
n_relay: "Outbox relay" {
  shape: rectangle
  style.multiple: true
}

n_client -> n_orders: "call"
n_orders -> n_odb: "call"
n_bus -> n_payments: "duplicate" {
  style.stroke-width: 4
}
n_payments -> n_bus: "async" {
  style.stroke-dash: 3
}
n_bus -> n_inventory: "async"
n_relay -> n_odb: "call"
n_relay -> n_bus: "duplicate" {
  style.stroke-width: 4
}

PlantUML: request sequence

@startuml
participant "Shop app" as n_client
participant "Orders" as n_orders
participant "Inventory" as n_inventory
participant "Payments" as n_payments
database "Orders DB" as n_odb
queue "Event log" as n_bus
note over n_client, n_orders : One order, three databases
n_client -> n_orders : POST /orders HTTP/1.1
n_orders --> n_client : HTTP/1.1 202 Accepted
note over n_orders, n_inventory : Money taken, stock call failed
n_orders -> n_inventory : POST /reservations HTTP/1.1
n_inventory --> n_orders : HTTP/1.1 503 Service Unavailable
note over n_orders, n_payments : Two-phase commit: everyone votes first
n_orders -> n_payments : PREPARE tx=ord_1042
n_payments --> n_orders : VOTE yes
note over n_orders, n_payments : The coordinator dies after the vote
n_orders ->x n_payments : [lost] PREPARE tx=ord_1042 -> VOTE yes
note over n_orders, n_odb : Saga: a chain of local transactions
n_orders -> n_odb : INSERT INTO sagas (id, order_id, step, status)
n_odb --> n_orders : UPDATE 1
note over n_orders, n_payments : Failure: undo with a compensating action
n_orders -> n_payments : POST /charges/ch_88/refunds HTTP/1.1
n_payments --> n_orders : HTTP/1.1 201 Created
note over n_orders, n_bus : Choreography: services react to events
n_orders ->> n_bus : produce topic=orders key=1042
n_bus -->> n_orders : ack partition=3 offset=7713
note over n_orders, n_bus : Saved in the DB, event never sent
n_orders ->x n_bus : [lost] produce topic=orders key=1042
note over n_orders, n_odb : Outbox: the event is written with the order
n_orders -> n_odb : BEGIN;
n_odb --> n_orders : COMMIT
note over n_bus, n_payments : At-least-once: consumers drop duplicates
n_bus ->> n_payments : [duplicate] deliver topic=orders key=1042 offset=7714
n_payments -->> n_bus : INSERT INTO processed (event_id) VALUES ('ev_5c1') ON CONFLICT DO NOTHING
@enduml

PlantUML: components

@startuml
left to right direction
agent "Shop app" as n_client
component "Orders" as n_orders
database "Orders DB" as n_odb
component "Payments" as n_payments
component "Inventory" as n_inventory
queue "Event log" as n_bus
collections "Outbox relay" as n_relay
n_client --> n_orders : call
n_orders --> n_odb : call
n_bus -[bold]-> n_payments : duplicate
n_payments ..> n_bus : async
n_bus --> n_inventory : async
n_relay --> n_odb : call
n_relay -[bold]-> n_bus : duplicate
@enduml

Recap

Four rules for work that spans services.

1) There is no transaction across services; avoid 2PC, it blocks everyone on the slowest node. 2) Use a saga: local steps, each with a compensating action, run by an orchestrator or by events. 3) Write events through an outbox in the same transaction as the data. 4) Delivery is at least once, so every consumer dedupes by event_id.

Numbers: at the start and at the end

MetricStartEnd
Order p95320 ms190 ms
Charged, no order0/day0/day
Lost events0/day0/day

Steps

  1. Case One order, three databases

    The shop app places an order.

  2. Problem Money taken, stock call failed

    Payments charged the card.

  3. Case Two-phase commit: everyone votes first

    A coordinator (here Orders) asks every participant to prepare: do the work, lock the rows, and vote.

  4. Problem The coordinator dies after the vote

    Orders crashed between the votes and the decision.

  5. Fix Saga: a chain of local transactions

    Drop the global lock.

  6. Fix Failure: undo with a compensating action

    Inventory answers "out of stock".

  7. Case Choreography: services react to events

    Another way to run the same saga: no orchestrator.

  8. Problem Saved in the DB, event never sent

    Orders commits the order and then publishes the event: two separate writes.

  9. Fix Outbox: the event is written with the order

    Orders writes the order and the event into an outbox table in one local transaction.

  10. Fix At-least-once: consumers drop duplicates

    The relay published row ev_5c1 and crashed before marking it sent.

  11. Recap Recap

    Four rules for work that spans services.

Terms

Two-phase commit (2PC)
A coordinator first asks every participant to prepare and vote, and sends commit only if all vote yes. Participants hold locks the whole time and get stuck if the coordinator dies, so teams avoid it.
Saga
Work across several services runs as a chain of local transactions, each with a compensating action. If a step fails, earlier steps are undone by their compensations; an orchestrator or events (choreography) drive the chain.
Outbox
A service writes the data and the event into an outbox table in one local transaction, and a separate relay publishes the event to the broker. The event can be late, but it can no longer be lost.
At-least-once delivery
A message is delivered at least once, and sometimes twice. So consumers drop duplicates by event_id or write idempotently.