Cheat sheet · Scaling
Load balancing
The problem
One balancer, three app servers. How the balancer chooses where each request goes, why the wrong choice leaves one server burning while the others idle, how it notices a dead server, and what sticky sessions really cost.
What you learn
- L4 vs L7: balancing connections vs balancing requests
- Round robin, least connections and weights, and when each one fails
- Health checks: how a dead server leaves the pool
- Why sticky sessions unbalance the load, and the stateless alternative
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_clients(["Clients"])
n_lb{"Load balancer"}
n_a["Server A"]
n_b["Server B"]
n_c["Server C"]
n_db[("Database")]
n_sessions{{"Session store"}}
n_clients --> n_lb
n_lb --> n_a
n_lb --> n_b
n_lb --> n_c
n_a --> n_db
n_b --> n_db
n_c --> n_db
n_a --> n_sessions
n_b --> n_sessions
n_c --> n_sessionsMermaid: request sequence
sequenceDiagram
participant n_lb as Load balancer
participant n_b as Server B
participant n_a as Server A
participant n_clients as Clients
participant n_c as Server C
participant n_sessions as Session store
Note over n_lb,n_b: An L4 balancer moves connections
n_lb->>n_b: TCP 198.51.100.7:52114 -#62; 192.0.2.12:8443
Note over n_lb,n_a: One busy connection, one burning server
n_lb->>n_a: balancer connection table
Note over n_lb,n_a: An L7 balancer moves requests
n_lb->>n_a: GET /api/orders?page=2 HTTP/1.1
Note over n_clients,n_lb: Round robin: take turns
n_clients->>n_lb: balancer access log
Note over n_lb,n_c: Same count, very different work
n_lb->>n_c: GET /api/reports/export?year=2026 HTTP/1.1
n_c-->>n_lb: HTTP/1.1 200 OK
Note over n_lb,n_b: Least connections: ask who is free
n_lb->>n_b: pick upstream for req_5e11
Note over n_lb,n_b: B dies, the balancer keeps sending
n_lb-xn_b: GET /api/orders HTTP/1.1
Note over n_lb,n_b: Health checks remove a dead server
n_lb->>n_b: GET /healthz HTTP/1.1
n_b-->>n_lb: timeout after 1000 ms
Note over n_clients,n_lb: Sticky sessions pin users to a server
n_clients->>n_lb: POST /api/cart/items HTTP/1.1
n_lb-->>n_clients: HTTP/1.1 200 OK
Note over n_b,n_sessions: Shared session store, no stickiness
n_b->>n_sessions: HGETALL session:9c1e7a
n_sessions-->>n_b: 3 fieldsD2
direction: right
n_clients: "Clients" {
shape: rectangle
}
n_lb: "Load balancer" {
shape: diamond
}
n_a: "Server A" {
shape: rectangle
}
n_b: "Server B" {
shape: rectangle
}
n_c: "Server C" {
shape: rectangle
}
n_db: "Database" {
shape: cylinder
}
n_sessions: "Session store" {
shape: hexagon
}
n_clients -> n_lb
n_lb -> n_a
n_lb -> n_b
n_lb -> n_c
n_a -> n_db
n_b -> n_db
n_c -> n_db
n_a -> n_sessions
n_b -> n_sessions
n_c -> n_sessionsPlantUML: request sequence
@startuml
participant "Load balancer" as n_lb
participant "Server B" as n_b
participant "Server A" as n_a
participant "Clients" as n_clients
participant "Server C" as n_c
collections "Session store" as n_sessions
note over n_lb, n_b : An L4 balancer moves connections
n_lb -> n_b : TCP 198.51.100.7:52114 -> 192.0.2.12:8443
note over n_lb, n_a : One busy connection, one burning server
n_lb -> n_a : balancer connection table
note over n_lb, n_a : An L7 balancer moves requests
n_lb -> n_a : GET /api/orders?page=2 HTTP/1.1
note over n_clients, n_lb : Round robin: take turns
n_clients -> n_lb : balancer access log
note over n_lb, n_c : Same count, very different work
n_lb -> n_c : GET /api/reports/export?year=2026 HTTP/1.1
n_c --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_b : Least connections: ask who is free
n_lb -> n_b : pick upstream for req_5e11
note over n_lb, n_b : B dies, the balancer keeps sending
n_lb ->x n_b : [lost] GET /api/orders HTTP/1.1
note over n_lb, n_b : Health checks remove a dead server
n_lb -> n_b : GET /healthz HTTP/1.1
n_b --> n_lb : timeout after 1000 ms
note over n_clients, n_lb : Sticky sessions pin users to a server
n_clients -> n_lb : POST /api/cart/items HTTP/1.1
n_lb --> n_clients : HTTP/1.1 200 OK
note over n_b, n_sessions : Shared session store, no stickiness
n_b -> n_sessions : HGETALL session:9c1e7a
n_sessions --> n_b : 3 fields
@endumlPlantUML: components
@startuml
left to right direction
agent "Clients" as n_clients
node "Load balancer" as n_lb
component "Server A" as n_a
component "Server B" as n_b
component "Server C" as n_c
database "Database" as n_db
stack "Session store" as n_sessions
n_clients --> n_lb
n_lb --> n_a
n_lb --> n_b
n_lb --> n_c
n_a --> n_db
n_b --> n_db
n_c --> n_db
n_a --> n_sessions
n_b --> n_sessions
n_c --> n_sessions
@endumlRecap
A balancer is only as good as its choices.
1) L4 balances connections, L7 balances requests: long-lived connections need L7. 2) Round robin for equal work, least connections for uneven work, weights for unequal servers. 3) Health checks take dead servers out in seconds. 4) Sticky sessions trade balance and resilience for convenience; shared state is the better fix.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Requests | 900/s | 1 300/s |
| Latency p99 | 140 ms | 160 ms |
| Errors | 0% | 0.3% |
| Busiest server | 35% | 55% |
Steps
- Case An L4 balancer moves connections
The balancer works at the transport layer (L4).
- Problem One busy connection, one burning server
A partner integration opens a single keep-alive connection and sends 400 requests a second through it.
- Fix An L7 balancer moves requests
The balancer now terminates TLS and speaks HTTP (L7).
- Case Round robin: take turns
The simplest algorithm: A, B, C, A, B, C… Each server gets the same number of requests.
- Problem Same count, very different work
A new report export takes 4 seconds per request; a normal request takes 13 ms.
- Fix Least connections: ask who is free
The balancer now tracks how many requests each server is working on right now and sends the next one to the server with the fewest.
- Case Weights for unequal servers
C is replaced by a machine twice as large.
- Problem B dies, the balancer keeps sending
Server B runs out of memory and crashes.
- Fix Health checks remove a dead server
Every 2 seconds the balancer calls
/healthzon each server. - Problem Sticky sessions pin users to a server
The cart lives in each server's memory, so the team turns on sticky sessions: the balancer sets a cookie and keeps sending each user to the same server.
- Fix Shared session store, no stickiness
Carts and sessions move to a shared store that every server reads.
- Recap Recap
A balancer is only as good as its choices.
Terms
- Load balancer
- A component that spreads incoming traffic across several servers. L4 balances connections, L7 balances requests; it picks by round robin, least connections or weights.
- Health check
- The balancer probes each server regularly and takes one that stops answering out of the pool within seconds. It comes back once it recovers.