Cheat sheet · Communication
Pub/sub and event logs
The problem
One event, many teams that need it. How a topic decouples them, and what offsets, consumer groups, keys and schemas add.
What you learn
- Why one slow subscriber should never break the checkout
- Offsets: how a consumer that was down catches up and replays
- Consumer groups and partitions: scaling one subscriber
- Ordering per key and changing an event without breaking readers
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_shop(["Shop app"])
n_orders["Orders"]
n_email("Email")
n_stock("Stock")
n_analytics("Analytics")
n_topic[["Topic"]]
n_stock_2("Stock 2")
n_shop --> n_orders
n_orders --> n_topic
n_topic --> n_email
n_topic --> n_stock
n_topic --> n_analytics
n_topic --> n_stock_2Mermaid: request sequence
sequenceDiagram
participant n_orders as Orders
participant n_email as Email
participant n_analytics as Analytics
participant n_topic as Topic
participant n_stock_2 as Stock 2
Note over n_orders,n_email: Orders calls everyone itself
n_orders->>n_email: POST /send-receipt HTTP/1.1
n_email-->>n_orders: HTTP/1.1 200 OK
Note over n_orders,n_analytics: Analytics is down, checkout is down
n_orders-xn_analytics: POST /track HTTP/1.1
Note over n_orders,n_topic: Publish one event to a topic
n_orders->>n_topic: PRODUCE topic=order-events
n_topic-->>n_orders: OK partition=2 offset=88412
Note over n_topic,n_analytics: A dead subscriber only falls behind
n_topic->>n_analytics: describe consumer group: analytics
n_analytics-->>n_topic: PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
Note over n_topic,n_analytics: Catch up from the saved offset
n_topic->>n_analytics: FETCH topic=order-events partition=2 offset=76112
n_analytics-->>n_topic: RECORDS 500 (offsets 76112..76611)
Note over n_topic,n_stock_2: A consumer group splits the partitions
n_topic->>n_stock_2: JOIN group=stock member=stock-2
n_stock_2-->>n_topic: REBALANCE group=stock generation=7
Note over n_orders,n_topic: Cancel arrives before the order
n_orders->>n_topic: PRODUCE topic=order-events
n_topic-->>n_orders: OK partition=5 offset=31907
Note over n_orders,n_topic: Same key, same partition
n_orders->>n_topic: PRODUCE topic=order-events
n_topic-->>n_orders: OK partition=2 offset=88950
Note over n_topic,n_analytics: A renamed field stops a reader
n_topic->>n_analytics: RECORD partition=2 offset=90233
n_analytics-->>n_topic: consumer stopped: field total is missing
Note over n_orders,n_topic: Version the event, change it additively
n_orders->>n_topic: PRODUCE topic=order-events
n_topic-->>n_orders: OK partition=0 offset=41022D2
direction: right
n_shop: "Shop app" {
shape: rectangle
}
n_orders: "Orders" {
shape: rectangle
}
n_email: "Email" {
shape: rectangle
style.multiple: true
}
n_stock: "Stock" {
shape: rectangle
style.multiple: true
}
n_analytics: "Analytics" {
shape: rectangle
style.multiple: true
}
n_topic: "Topic" {
shape: queue
}
n_stock_2: "Stock 2" {
shape: rectangle
style.multiple: true
}
n_shop -> n_orders
n_orders -> n_topic
n_topic -> n_email
n_topic -> n_stock
n_topic -> n_analytics
n_topic -> n_stock_2PlantUML: request sequence
@startuml
participant "Orders" as n_orders
participant "Email" as n_email
participant "Analytics" as n_analytics
queue "Topic" as n_topic
participant "Stock 2" as n_stock_2
note over n_orders, n_email : Orders calls everyone itself
n_orders -> n_email : POST /send-receipt HTTP/1.1
n_email --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_analytics : Analytics is down, checkout is down
n_orders ->x n_analytics : [lost] POST /track HTTP/1.1
note over n_orders, n_topic : Publish one event to a topic
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=2 offset=88412
note over n_topic, n_analytics : A dead subscriber only falls behind
n_topic -> n_analytics : describe consumer group: analytics
n_analytics --> n_topic : PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
note over n_topic, n_analytics : Catch up from the saved offset
n_topic -> n_analytics : FETCH topic=order-events partition=2 offset=76112
n_analytics --> n_topic : RECORDS 500 (offsets 76112..76611)
note over n_topic, n_stock_2 : A consumer group splits the partitions
n_topic -> n_stock_2 : JOIN group=stock member=stock-2
n_stock_2 --> n_topic : REBALANCE group=stock generation=7
note over n_orders, n_topic : Cancel arrives before the order
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=5 offset=31907
note over n_orders, n_topic : Same key, same partition
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=2 offset=88950
note over n_topic, n_analytics : A renamed field stops a reader
n_topic -> n_analytics : RECORD partition=2 offset=90233
n_analytics --> n_topic : consumer stopped: field total is missing
note over n_orders, n_topic : Version the event, change it additively
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=0 offset=41022
@endumlPlantUML: components
@startuml
left to right direction
agent "Shop app" as n_shop
component "Orders" as n_orders
collections "Email" as n_email
collections "Stock" as n_stock
collections "Analytics" as n_analytics
queue "Topic" as n_topic
collections "Stock 2" as n_stock_2
n_shop --> n_orders
n_orders --> n_topic
n_topic --> n_email
n_topic --> n_stock
n_topic --> n_analytics
n_topic --> n_stock_2
@endumlRecap
The producer writes a fact once; each subscriber group reads it at its own pace through its own offset.
1) Lag, not errors, shows a slow subscriber. 2) Commit offsets after work, and replay when needed. 3) Scale a subscriber with a group, up to the partition count. 4) Order exists only per key. 5) Events are contracts: version them and change them additively.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Checkout p95 | 900 ms | 40 ms |
| Failed orders | 0% | 0% |
| Lag (events) | 0 | 0 |
| Out of order | 0/min | 0/min |
Steps
- Problem Orders calls everyone itself
When an order is placed, the orders service calls email, stock and analytics one by one, and only then answers the shop.
- Problem Analytics is down, checkout is down
The analytics service hangs.
- Fix Publish one event to a topic
Orders now writes one event,
order.placed, to a topic and answers the shop. - Case A dead subscriber only falls behind
Analytics is down again.
- Fix Catch up from the saved offset
Analytics comes back and asks for records from its last committed offset.
- Problem One subscriber cannot keep up
A sale triples the number of orders.
- Fix A consumer group splits the partitions
The topic has 6 partitions.
- Problem Cancel arrives before the order
A customer orders and cancels a second later.
- Fix Same key, same partition
The producer now sets the record key to the order ID.
- Problem A renamed field stops a reader
The orders team renames
totaltoamountand deploys. - Fix Version the event, change it additively
Events get a schema with a version, checked when they are produced.
- Recap Recap
The producer writes a fact once; each subscriber group reads it at its own pace through its own offset.
Terms
- Pub/sub
- A producer writes an event to a topic once, and each subscriber group reads it at its own pace through its own offset. A slow or dead subscriber only falls behind; it does not stop the producer.
- Consumer group
- Several instances of one subscriber split the topic's partitions, and each partition is read by exactly one member. It scales up to the partition count, and order holds only per key.