Cheat sheet · Whole systems
Design a chat
The problem
Alice writes to Bob, and the message should appear on his screen in a fraction of a second, in the right order, even if his phone was asleep. We build the chat step by step: WebSocket gateways, routing, storage by conversation, sequence numbers, receipts, push, groups and presence.
What you learn
- Why a chat keeps a WebSocket open and how a message finds the right gateway
- Why messages are partitioned by conversation, not by user
- Per-conversation sequence numbers instead of clocks
- Receipts, push for offline users, group fan-out and presence with TTL
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_alice(["Alice's app"])
n_bob(["Bob's app"])
n_chat["Chat service"]
n_msgs[("Message store")]
n_gw_a["Gateway A"]
n_gw_b["Gateway B"]
n_sessions{{"Session registry"}}
n_notify("Push worker")
n_push>"Push service"]
n_chat --> n_msgs
n_alice --> n_gw_a
n_gw_a --> n_chat
n_gw_b --> n_sessions
n_chat --> n_sessions
n_chat --> n_gw_b
n_chat -.-> n_gw_a
n_gw_a -.-> n_alice
n_chat --> n_notify
n_notify --> n_push
n_push --> n_bob
n_bob --> n_gw_b
n_gw_b --> n_bobMermaid: request sequence
sequenceDiagram
participant n_bob as Bob's app
participant n_chat as Chat service
participant n_gw_b as Gateway B
participant n_sessions as Session registry
participant n_msgs as Message store
participant n_notify as Push worker
participant n_push as Push service
Note over n_bob,n_chat: Asking every two seconds: anything new?
n_bob->>n_chat: GET /v1/messages?since=1795400100 HTTP/1.1
n_chat-->>n_bob: HTTP/1.1 200 OK
Note over n_bob,n_gw_b: One open connection per device
n_bob->>n_gw_b: GET /ws HTTP/1.1
n_gw_b-->>n_bob: HTTP/1.1 101 Switching Protocols
Note over n_chat,n_sessions: Finding Bob's gateway
n_chat->>n_sessions: SMEMBERS conn:u_bob
n_sessions-->>n_chat: 1) #34;gw-b/c_8812#34;
Note over n_chat,n_msgs: Store by conversation, not by user
n_chat->>n_msgs: INSERT INTO messages
n_msgs-->>n_chat: INSERT 0 1 -- shard 12 of 64
Note over n_gw_b,n_bob: Two quick messages arrive swapped
n_gw_b->>n_bob: u_bob history, sorted by sent_at:
Note over n_chat,n_msgs: A sequence number per conversation
n_chat->>n_msgs: UPDATE conversations SET last_seq = last_seq + 1
n_msgs-->>n_chat: last_seq = 1043
Note over n_bob,n_gw_b: Delivered and read
n_bob->>n_gw_b: WS frame (client -#62; server)
Note over n_chat,n_sessions: Bob's phone is asleep
n_chat->>n_sessions: SMEMBERS conn:u_bob
n_sessions-->>n_chat: (empty array)
Note over n_notify,n_push: Push notification, then catch-up
n_notify->>n_push: POST /v1/send HTTP/1.1
n_push-->>n_notify: HTTP/1.1 200 OK
Note over n_chat,n_sessions: Group chat: store once, fan out to who is online
n_chat->>n_sessions: SINTER members:g_team online
n_sessions-->>n_chat: 1) #34;u_bob#34;
Note over n_gw_b,n_sessions: Online and last seen
n_gw_b->>n_sessions: SET presence:u_bob online EX 60
n_sessions-->>n_gw_b: OKD2
direction: right
n_alice: "Alice's app" {
shape: rectangle
}
n_bob: "Bob's app" {
shape: rectangle
}
n_chat: "Chat service" {
shape: rectangle
}
n_msgs: "Message store" {
shape: cylinder
}
n_gw_a: "Gateway A" {
shape: rectangle
}
n_gw_b: "Gateway B" {
shape: rectangle
}
n_sessions: "Session registry" {
shape: hexagon
}
n_notify: "Push worker" {
shape: rectangle
style.multiple: true
}
n_push: "Push service" {
shape: cloud
}
n_chat -> n_msgs
n_alice -> n_gw_a
n_gw_a -> n_chat
n_gw_b -> n_sessions
n_chat -> n_sessions
n_chat -> n_gw_b
n_chat -> n_gw_a {
style.stroke-dash: 3
}
n_gw_a -> n_alice {
style.stroke-dash: 3
}
n_chat -> n_notify
n_notify -> n_push
n_push -> n_bob
n_bob -> n_gw_b
n_gw_b -> n_bobPlantUML: request sequence
@startuml
participant "Bob's app" as n_bob
participant "Chat service" as n_chat
participant "Gateway B" as n_gw_b
collections "Session registry" as n_sessions
database "Message store" as n_msgs
participant "Push worker" as n_notify
boundary "Push service" as n_push
note over n_bob, n_chat : Asking every two seconds: anything new?
n_bob -> n_chat : GET /v1/messages?since=1795400100 HTTP/1.1
n_chat --> n_bob : HTTP/1.1 200 OK
note over n_bob, n_gw_b : One open connection per device
n_bob -> n_gw_b : GET /ws HTTP/1.1
n_gw_b --> n_bob : HTTP/1.1 101 Switching Protocols
note over n_chat, n_sessions : Finding Bob's gateway
n_chat -> n_sessions : SMEMBERS conn:u_bob
n_sessions --> n_chat : 1) "gw-b/c_8812"
note over n_chat, n_msgs : Store by conversation, not by user
n_chat -> n_msgs : INSERT INTO messages
n_msgs --> n_chat : INSERT 0 1 -- shard 12 of 64
note over n_gw_b, n_bob : Two quick messages arrive swapped
n_gw_b -> n_bob : u_bob history, sorted by sent_at:
note over n_chat, n_msgs : A sequence number per conversation
n_chat -> n_msgs : UPDATE conversations SET last_seq = last_seq + 1
n_msgs --> n_chat : last_seq = 1043
note over n_bob, n_gw_b : Delivered and read
n_bob -> n_gw_b : WS frame (client -> server)
note over n_chat, n_sessions : Bob's phone is asleep
n_chat -> n_sessions : SMEMBERS conn:u_bob
n_sessions --> n_chat : (empty array)
note over n_notify, n_push : Push notification, then catch-up
n_notify -> n_push : POST /v1/send HTTP/1.1
n_push --> n_notify : HTTP/1.1 200 OK
note over n_chat, n_sessions : Group chat: store once, fan out to who is online
n_chat -> n_sessions : SINTER members:g_team online
n_sessions --> n_chat : 1) "u_bob"
note over n_gw_b, n_sessions : Online and last seen
n_gw_b -> n_sessions : SET presence:u_bob online EX 60
n_sessions --> n_gw_b : OK
@endumlPlantUML: components
@startuml
left to right direction
agent "Alice's app" as n_alice
agent "Bob's app" as n_bob
component "Chat service" as n_chat
database "Message store" as n_msgs
component "Gateway A" as n_gw_a
component "Gateway B" as n_gw_b
stack "Session registry" as n_sessions
collections "Push worker" as n_notify
cloud "Push service" as n_push
n_chat --> n_msgs
n_alice --> n_gw_a
n_gw_a --> n_chat
n_gw_b --> n_sessions
n_chat --> n_sessions
n_chat --> n_gw_b
n_chat ..> n_gw_a
n_gw_a ..> n_alice
n_chat --> n_notify
n_notify --> n_push
n_push --> n_bob
n_bob --> n_gw_b
n_gw_b --> n_bob
@endumlRecap
A chat is a store plus a set of open pipes.
1) WebSocket gateways hold connections; a registry tells where each user is. 2) Store first, partitioned by conversation. 3) Order by a per-conversation sequence, never by clocks; clients fill gaps by seq. 4) Receipts are one number per member. 5) Offline users get a push and catch up from the store. 6) Groups fan out per gateway, presence lives on a TTL.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Delivery p99 | 2 100 ms | 180 ms |
| Out-of-order messages | 0/day | 0/day |
| Unnoticed for 1 h+ | 0/day | 15/day |
Steps
- Problem Asking every two seconds: anything new?
The first version: Bob's app asks the server every 2 seconds for new messages.
- Fix One open connection per device
Each app opens a WebSocket to a gateway and keeps it open.
- Case Finding Bob's gateway
Alice's message arrives at gateway A, which passes it to the chat service.
- Case Store by conversation, not by user
Every message is written before it is delivered.
- Problem Two quick messages arrive swapped
Alice sends "dinner tonight?" and right after it "at 7pm?".
- Fix A sequence number per conversation
The shard that owns the conversation hands out the next number atomically: 1042, 1043, 1044.
- Case Delivered and read
When Bob's device gets the message it acknowledges it: one tick for Alice.
- Problem Bob's phone is asleep
The phone puts the app in the background and the operating system closes its socket.
- Fix Push notification, then catch-up
When no connection is found, the chat service puts an event on a queue.
- Case Group chat: store once, fan out to who is online
A team chat with 300 members.
- Case Online and last seen
Each gateway sends a heartbeat for its users every 25 s, and the presence key lives for 60 s.
- Recap Recap
A chat is a store plus a set of open pipes.