mufassall

Cheat sheet · Communication

Pub/sub and event logs

13 min · Intermediate · mufassall.dtwin.uz/en/l/pubsub/

← Open the lesson

The problem

One event, many teams that need it. How a topic decouples them, and what offsets, consumer groups, keys and schemas add.

What you learn

  • Why one slow subscriber should never break the checkout
  • Offsets: how a consumer that was down catches up and replays
  • Consumer groups and partitions: scaling one subscriber
  • Ordering per key and changing an event without breaking readers
Final scene
55%50%25%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_shop(["Shop app"])
  n_orders["Orders"]
  n_email("Email")
  n_stock("Stock")
  n_analytics("Analytics")
  n_topic[["Topic"]]
  n_stock_2("Stock 2")
  n_shop --> n_orders
  n_orders --> n_topic
  n_topic --> n_email
  n_topic --> n_stock
  n_topic --> n_analytics
  n_topic --> n_stock_2

Mermaid: request sequence

sequenceDiagram
  participant n_orders as Orders
  participant n_email as Email
  participant n_analytics as Analytics
  participant n_topic as Topic
  participant n_stock_2 as Stock 2
  Note over n_orders,n_email: Orders calls everyone itself
  n_orders->>n_email: POST /send-receipt HTTP/1.1
  n_email-->>n_orders: HTTP/1.1 200 OK
  Note over n_orders,n_analytics: Analytics is down, checkout is down
  n_orders-xn_analytics: POST /track HTTP/1.1
  Note over n_orders,n_topic: Publish one event to a topic
  n_orders->>n_topic: PRODUCE topic=order-events
  n_topic-->>n_orders: OK partition=2 offset=88412
  Note over n_topic,n_analytics: A dead subscriber only falls behind
  n_topic->>n_analytics: describe consumer group: analytics
  n_analytics-->>n_topic: PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
  Note over n_topic,n_analytics: Catch up from the saved offset
  n_topic->>n_analytics: FETCH topic=order-events partition=2 offset=76112
  n_analytics-->>n_topic: RECORDS 500 (offsets 76112..76611)
  Note over n_topic,n_stock_2: A consumer group splits the partitions
  n_topic->>n_stock_2: JOIN group=stock member=stock-2
  n_stock_2-->>n_topic: REBALANCE group=stock generation=7
  Note over n_orders,n_topic: Cancel arrives before the order
  n_orders->>n_topic: PRODUCE topic=order-events
  n_topic-->>n_orders: OK partition=5 offset=31907
  Note over n_orders,n_topic: Same key, same partition
  n_orders->>n_topic: PRODUCE topic=order-events
  n_topic-->>n_orders: OK partition=2 offset=88950
  Note over n_topic,n_analytics: A renamed field stops a reader
  n_topic->>n_analytics: RECORD partition=2 offset=90233
  n_analytics-->>n_topic: consumer stopped: field total is missing
  Note over n_orders,n_topic: Version the event, change it additively
  n_orders->>n_topic: PRODUCE topic=order-events
  n_topic-->>n_orders: OK partition=0 offset=41022

D2

direction: right

n_shop: "Shop app" {
  shape: rectangle
}
n_orders: "Orders" {
  shape: rectangle
}
n_email: "Email" {
  shape: rectangle
  style.multiple: true
}
n_stock: "Stock" {
  shape: rectangle
  style.multiple: true
}
n_analytics: "Analytics" {
  shape: rectangle
  style.multiple: true
}
n_topic: "Topic" {
  shape: queue
}
n_stock_2: "Stock 2" {
  shape: rectangle
  style.multiple: true
}

n_shop -> n_orders
n_orders -> n_topic
n_topic -> n_email
n_topic -> n_stock
n_topic -> n_analytics
n_topic -> n_stock_2

PlantUML: request sequence

@startuml
participant "Orders" as n_orders
participant "Email" as n_email
participant "Analytics" as n_analytics
queue "Topic" as n_topic
participant "Stock 2" as n_stock_2
note over n_orders, n_email : Orders calls everyone itself
n_orders -> n_email : POST /send-receipt HTTP/1.1
n_email --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_analytics : Analytics is down, checkout is down
n_orders ->x n_analytics : [lost] POST /track HTTP/1.1
note over n_orders, n_topic : Publish one event to a topic
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=2 offset=88412
note over n_topic, n_analytics : A dead subscriber only falls behind
n_topic -> n_analytics : describe consumer group: analytics
n_analytics --> n_topic : PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
note over n_topic, n_analytics : Catch up from the saved offset
n_topic -> n_analytics : FETCH topic=order-events partition=2 offset=76112
n_analytics --> n_topic : RECORDS 500 (offsets 76112..76611)
note over n_topic, n_stock_2 : A consumer group splits the partitions
n_topic -> n_stock_2 : JOIN group=stock member=stock-2
n_stock_2 --> n_topic : REBALANCE group=stock generation=7
note over n_orders, n_topic : Cancel arrives before the order
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=5 offset=31907
note over n_orders, n_topic : Same key, same partition
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=2 offset=88950
note over n_topic, n_analytics : A renamed field stops a reader
n_topic -> n_analytics : RECORD partition=2 offset=90233
n_analytics --> n_topic : consumer stopped: field total is missing
note over n_orders, n_topic : Version the event, change it additively
n_orders -> n_topic : PRODUCE topic=order-events
n_topic --> n_orders : OK partition=0 offset=41022
@enduml

PlantUML: components

@startuml
left to right direction
agent "Shop app" as n_shop
component "Orders" as n_orders
collections "Email" as n_email
collections "Stock" as n_stock
collections "Analytics" as n_analytics
queue "Topic" as n_topic
collections "Stock 2" as n_stock_2
n_shop --> n_orders
n_orders --> n_topic
n_topic --> n_email
n_topic --> n_stock
n_topic --> n_analytics
n_topic --> n_stock_2
@enduml

Recap

The producer writes a fact once; each subscriber group reads it at its own pace through its own offset.

1) Lag, not errors, shows a slow subscriber. 2) Commit offsets after work, and replay when needed. 3) Scale a subscriber with a group, up to the partition count. 4) Order exists only per key. 5) Events are contracts: version them and change them additively.

Numbers: at the start and at the end

MetricStartEnd
Checkout p95900 ms40 ms
Failed orders0%0%
Lag (events)00
Out of order0/min0/min

Steps

  1. Problem Orders calls everyone itself

    When an order is placed, the orders service calls email, stock and analytics one by one, and only then answers the shop.

  2. Problem Analytics is down, checkout is down

    The analytics service hangs.

  3. Fix Publish one event to a topic

    Orders now writes one event, order.placed, to a topic and answers the shop.

  4. Case A dead subscriber only falls behind

    Analytics is down again.

  5. Fix Catch up from the saved offset

    Analytics comes back and asks for records from its last committed offset.

  6. Problem One subscriber cannot keep up

    A sale triples the number of orders.

  7. Fix A consumer group splits the partitions

    The topic has 6 partitions.

  8. Problem Cancel arrives before the order

    A customer orders and cancels a second later.

  9. Fix Same key, same partition

    The producer now sets the record key to the order ID.

  10. Problem A renamed field stops a reader

    The orders team renames total to amount and deploys.

  11. Fix Version the event, change it additively

    Events get a schema with a version, checked when they are produced.

  12. Recap Recap

    The producer writes a fact once; each subscriber group reads it at its own pace through its own offset.

Terms

Pub/sub
A producer writes an event to a topic once, and each subscriber group reads it at its own pace through its own offset. A slow or dead subscriber only falls behind; it does not stop the producer.
Consumer group
Several instances of one subscriber split the topic's partitions, and each partition is read by exactly one member. It scales up to the partition count, and order holds only per key.