mufassall

Cheat sheet · Scaling

Load balancing

13 min · Intermediate · mufassall.dtwin.uz/en/l/load-balancing/

← Open the lesson

The problem

One balancer, three app servers. How the balancer chooses where each request goes, why the wrong choice leaves one server burning while the others idle, how it notices a dead server, and what sticky sessions really cost.

What you learn

  • L4 vs L7: balancing connections vs balancing requests
  • Round robin, least connections and weights, and when each one fails
  • Health checks: how a dead server leaves the pool
  • Why sticky sessions unbalance the load, and the stateless alternative
Final scene
55%55%55%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_clients(["Clients"])
  n_lb{"Load balancer"}
  n_a["Server A"]
  n_b["Server B"]
  n_c["Server C"]
  n_db[("Database")]
  n_sessions{{"Session store"}}
  n_clients --> n_lb
  n_lb --> n_a
  n_lb --> n_b
  n_lb --> n_c
  n_a --> n_db
  n_b --> n_db
  n_c --> n_db
  n_a --> n_sessions
  n_b --> n_sessions
  n_c --> n_sessions

Mermaid: request sequence

sequenceDiagram
  participant n_lb as Load balancer
  participant n_b as Server B
  participant n_a as Server A
  participant n_clients as Clients
  participant n_c as Server C
  participant n_sessions as Session store
  Note over n_lb,n_b: An L4 balancer moves connections
  n_lb->>n_b: TCP 198.51.100.7:52114 -#62; 192.0.2.12:8443
  Note over n_lb,n_a: One busy connection, one burning server
  n_lb->>n_a: balancer connection table
  Note over n_lb,n_a: An L7 balancer moves requests
  n_lb->>n_a: GET /api/orders?page=2 HTTP/1.1
  Note over n_clients,n_lb: Round robin: take turns
  n_clients->>n_lb: balancer access log
  Note over n_lb,n_c: Same count, very different work
  n_lb->>n_c: GET /api/reports/export?year=2026 HTTP/1.1
  n_c-->>n_lb: HTTP/1.1 200 OK
  Note over n_lb,n_b: Least connections: ask who is free
  n_lb->>n_b: pick upstream for req_5e11
  Note over n_lb,n_b: B dies, the balancer keeps sending
  n_lb-xn_b: GET /api/orders HTTP/1.1
  Note over n_lb,n_b: Health checks remove a dead server
  n_lb->>n_b: GET /healthz HTTP/1.1
  n_b-->>n_lb: timeout after 1000 ms
  Note over n_clients,n_lb: Sticky sessions pin users to a server
  n_clients->>n_lb: POST /api/cart/items HTTP/1.1
  n_lb-->>n_clients: HTTP/1.1 200 OK
  Note over n_b,n_sessions: Shared session store, no stickiness
  n_b->>n_sessions: HGETALL session:9c1e7a
  n_sessions-->>n_b: 3 fields

D2

direction: right

n_clients: "Clients" {
  shape: rectangle
}
n_lb: "Load balancer" {
  shape: diamond
}
n_a: "Server A" {
  shape: rectangle
}
n_b: "Server B" {
  shape: rectangle
}
n_c: "Server C" {
  shape: rectangle
}
n_db: "Database" {
  shape: cylinder
}
n_sessions: "Session store" {
  shape: hexagon
}

n_clients -> n_lb
n_lb -> n_a
n_lb -> n_b
n_lb -> n_c
n_a -> n_db
n_b -> n_db
n_c -> n_db
n_a -> n_sessions
n_b -> n_sessions
n_c -> n_sessions

PlantUML: request sequence

@startuml
participant "Load balancer" as n_lb
participant "Server B" as n_b
participant "Server A" as n_a
participant "Clients" as n_clients
participant "Server C" as n_c
collections "Session store" as n_sessions
note over n_lb, n_b : An L4 balancer moves connections
n_lb -> n_b : TCP 198.51.100.7:52114 -> 192.0.2.12:8443
note over n_lb, n_a : One busy connection, one burning server
n_lb -> n_a : balancer connection table
note over n_lb, n_a : An L7 balancer moves requests
n_lb -> n_a : GET /api/orders?page=2 HTTP/1.1
note over n_clients, n_lb : Round robin: take turns
n_clients -> n_lb : balancer access log
note over n_lb, n_c : Same count, very different work
n_lb -> n_c : GET /api/reports/export?year=2026 HTTP/1.1
n_c --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_b : Least connections: ask who is free
n_lb -> n_b : pick upstream for req_5e11
note over n_lb, n_b : B dies, the balancer keeps sending
n_lb ->x n_b : [lost] GET /api/orders HTTP/1.1
note over n_lb, n_b : Health checks remove a dead server
n_lb -> n_b : GET /healthz HTTP/1.1
n_b --> n_lb : timeout after 1000 ms
note over n_clients, n_lb : Sticky sessions pin users to a server
n_clients -> n_lb : POST /api/cart/items HTTP/1.1
n_lb --> n_clients : HTTP/1.1 200 OK
note over n_b, n_sessions : Shared session store, no stickiness
n_b -> n_sessions : HGETALL session:9c1e7a
n_sessions --> n_b : 3 fields
@enduml

PlantUML: components

@startuml
left to right direction
agent "Clients" as n_clients
node "Load balancer" as n_lb
component "Server A" as n_a
component "Server B" as n_b
component "Server C" as n_c
database "Database" as n_db
stack "Session store" as n_sessions
n_clients --> n_lb
n_lb --> n_a
n_lb --> n_b
n_lb --> n_c
n_a --> n_db
n_b --> n_db
n_c --> n_db
n_a --> n_sessions
n_b --> n_sessions
n_c --> n_sessions
@enduml

Recap

A balancer is only as good as its choices.

1) L4 balances connections, L7 balances requests: long-lived connections need L7. 2) Round robin for equal work, least connections for uneven work, weights for unequal servers. 3) Health checks take dead servers out in seconds. 4) Sticky sessions trade balance and resilience for convenience; shared state is the better fix.

Numbers: at the start and at the end

MetricStartEnd
Requests900/s1 300/s
Latency p99140 ms160 ms
Errors0%0.3%
Busiest server35%55%

Steps

  1. Case An L4 balancer moves connections

    The balancer works at the transport layer (L4).

  2. Problem One busy connection, one burning server

    A partner integration opens a single keep-alive connection and sends 400 requests a second through it.

  3. Fix An L7 balancer moves requests

    The balancer now terminates TLS and speaks HTTP (L7).

  4. Case Round robin: take turns

    The simplest algorithm: A, B, C, A, B, C… Each server gets the same number of requests.

  5. Problem Same count, very different work

    A new report export takes 4 seconds per request; a normal request takes 13 ms.

  6. Fix Least connections: ask who is free

    The balancer now tracks how many requests each server is working on right now and sends the next one to the server with the fewest.

  7. Case Weights for unequal servers

    C is replaced by a machine twice as large.

  8. Problem B dies, the balancer keeps sending

    Server B runs out of memory and crashes.

  9. Fix Health checks remove a dead server

    Every 2 seconds the balancer calls /healthz on each server.

  10. Problem Sticky sessions pin users to a server

    The cart lives in each server's memory, so the team turns on sticky sessions: the balancer sets a cookie and keeps sending each user to the same server.

  11. Fix Shared session store, no stickiness

    Carts and sessions move to a shared store that every server reads.

  12. Recap Recap

    A balancer is only as good as its choices.

Terms

Load balancer
A component that spreads incoming traffic across several servers. L4 balances connections, L7 balances requests; it picks by round robin, least connections or weights.
Health check
The balancer probes each server regularly and takes one that stops answering out of the pool within seconds. It comes back once it recovers.