Cheat sheet · Scaling
Scaling up vs scaling out
The problem
A feed API outgrows its server. First it moves to a bigger box, until there is no bigger box and the one box falls over. Then it spreads over several smaller ones, and learns why that needs a load balancer and an app that remembers nothing.
What you learn
- Vertical scaling: why it is easy, and where it stops
- Why one server is a single point of failure, however big
- What horizontal scaling needs: a load balancer and stateless app servers
- Why you keep one server's worth of spare capacity (N+1)
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_users(["Users"])
n_app["App server 1"]
n_db[("Database")]
n_lb{"Load balancer"}
n_app2["App server 2"]
n_app3["App server 3"]
n_sessions{{"Session store"}}
n_app --> n_db
n_users --> n_lb
n_lb -.-> n_users
n_lb --> n_app
n_lb --x|"lost"| n_app2
n_lb --> n_app3
n_app2 --> n_db
n_app3 --> n_db
n_app --> n_sessions
n_app2 --> n_sessions
n_app3 --> n_sessions
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_app2 downMermaid: request sequence
sequenceDiagram
participant n_users as Users
participant n_app as App server 1
participant n_lb as Load balancer
participant n_app2 as App server 2
participant n_sessions as Session store
Note over n_users,n_app: One server does everything
n_users->>n_app: GET /feed HTTP/1.1
n_app-->>n_users: HTTP/1.1 200 OK
Note over n_users,n_app: Traffic doubles, the box is full
n_users->>n_app: GET /feed HTTP/1.1
n_app-->>n_users: HTTP/1.1 503 Service Unavailable
Note over n_users,n_app: One box, one failure
n_users-xn_app: curl https://api.example.com/feed
Note over n_lb,n_app2: Horizontal scaling: more boxes
n_lb->>n_app2: GET /feed HTTP/1.1
n_app2-->>n_lb: HTTP/1.1 200 OK
Note over n_lb,n_app2: Users keep getting logged out
n_lb->>n_app2: GET /cart HTTP/1.1
n_app2-->>n_lb: HTTP/1.1 302 Found
Note over n_app2,n_sessions: Make the app servers stateless
n_app2->>n_sessions: GET session:9c1e7a
n_sessions-->>n_app2: (bulk string)
Note over n_lb,n_app2: One server dies, users barely notice
n_lb-xn_app2: GET /healthz HTTP/1.1D2
direction: right
n_users: "Users" {
shape: rectangle
}
n_app: "App server 1" {
shape: rectangle
}
n_db: "Database" {
shape: cylinder
}
n_lb: "Load balancer" {
shape: diamond
}
n_app2: "App server 2" {
shape: rectangle
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_app3: "App server 3" {
shape: rectangle
}
n_sessions: "Session store" {
shape: hexagon
}
n_app -> n_db
n_users -> n_lb
n_lb -> n_users {
style.stroke-dash: 3
}
n_lb -> n_app
n_lb -> n_app2: "lost" {
target-arrowhead.shape: cross
}
n_lb -> n_app3
n_app2 -> n_db
n_app3 -> n_db
n_app -> n_sessions
n_app2 -> n_sessions
n_app3 -> n_sessionsPlantUML: request sequence
@startuml
participant "Users" as n_users
participant "App server 1" as n_app
participant "Load balancer" as n_lb
participant "App server 2" as n_app2
collections "Session store" as n_sessions
note over n_users, n_app : One server does everything
n_users -> n_app : GET /feed HTTP/1.1
n_app --> n_users : HTTP/1.1 200 OK
note over n_users, n_app : Traffic doubles, the box is full
n_users -> n_app : GET /feed HTTP/1.1
n_app --> n_users : HTTP/1.1 503 Service Unavailable
note over n_users, n_app : One box, one failure
n_users ->x n_app : [lost] curl https://api.example.com/feed
note over n_lb, n_app2 : Horizontal scaling: more boxes
n_lb -> n_app2 : GET /feed HTTP/1.1
n_app2 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : Users keep getting logged out
n_lb -> n_app2 : GET /cart HTTP/1.1
n_app2 --> n_lb : HTTP/1.1 302 Found
note over n_app2, n_sessions : Make the app servers stateless
n_app2 -> n_sessions : GET session:9c1e7a
n_sessions --> n_app2 : (bulk string)
note over n_lb, n_app2 : One server dies, users barely notice
n_lb ->x n_app2 : [lost] GET /healthz HTTP/1.1
@endumlPlantUML: components
@startuml
left to right direction
agent "Users" as n_users
component "App server 1" as n_app
database "Database" as n_db
node "Load balancer" as n_lb
component "App server 2" as n_app2 #fde2e2;line:c62828;line.dashed
component "App server 3" as n_app3
stack "Session store" as n_sessions
n_app --> n_db
n_users --> n_lb
n_lb ..> n_users
n_lb --> n_app
n_lb -[#c62828,dashed]-> n_app2 : lost
n_lb --> n_app3
n_app2 --> n_db
n_app3 --> n_db
n_app --> n_sessions
n_app2 --> n_sessions
n_app3 --> n_sessions
@endumlRecap
Scale up first: it is simple. Scale out when the box gets too expensive, too big, or too risky to be the only one.
1) One server, however big, is a single point of failure. 2) Many servers need a load balancer in front. 3) App servers must be stateless: sessions and files go to shared stores. 4) Keep N+1. 5) The database is still one box: it is the next thing to scale.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Served | 300/s | 2 400/s |
| Latency p95 | 80 ms | 240 ms |
| Errors | 0% | 0.5% |
| Cost per month | 160 $ | 1 040 $ |
Steps
- Case One server does everything
A small feed API runs on one 4-vCPU server.
- Problem Traffic doubles, the box is full
After a marketing push, users want 700 requests a second.
- Fix Vertical scaling: a bigger box
The easiest fix: move to a 16-vCPU server.
- Problem There is no bigger box
A year later: 2 400 requests a second.
- Problem One box, one failure
A failed disk takes the server down.
- Fix Horizontal scaling: more boxes
Instead of one 64-vCPU giant, three 8-vCPU servers.
- Problem Users keep getting logged out
Sessions were stored in the memory of the server where the user logged in.
- Fix Make the app servers stateless
Sessions move to a shared, fast session store.
- Case One server dies, users barely notice
app-2crashes. - Recap Recap
Scale up first: it is simple.
Terms
- Vertical scaling
- Replacing a server with a bigger, stronger one. It is simple, but one day there is no bigger box, and it is still a single point of failure.
- Horizontal scaling
- Adding more servers instead of one bigger one. It needs a load balancer in front and stateless app servers.