mufassall

Cheat sheet · Scaling

Scaling up vs scaling out

11 min · Beginner · mufassall.dtwin.uz/en/l/vertical-horizontal/

← Open the lesson

The problem

A feed API outgrows its server. First it moves to a bigger box, until there is no bigger box and the one box falls over. Then it spreads over several smaller ones, and learns why that needs a load balancer and an app that remembers nothing.

What you learn

  • Vertical scaling: why it is easy, and where it stops
  • Why one server is a single point of failure, however big
  • What horizontal scaling needs: a load balancer and stateless app servers
  • Why you keep one server's worth of spare capacity (N+1)
Final scene
92%92%70%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_users(["Users"])
  n_app["App server 1"]
  n_db[("Database")]
  n_lb{"Load balancer"}
  n_app2["App server 2"]
  n_app3["App server 3"]
  n_sessions{{"Session store"}}
  n_app --> n_db
  n_users --> n_lb
  n_lb -.-> n_users
  n_lb --> n_app
  n_lb --x|"lost"| n_app2
  n_lb --> n_app3
  n_app2 --> n_db
  n_app3 --> n_db
  n_app --> n_sessions
  n_app2 --> n_sessions
  n_app3 --> n_sessions
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_app2 down

Mermaid: request sequence

sequenceDiagram
  participant n_users as Users
  participant n_app as App server 1
  participant n_lb as Load balancer
  participant n_app2 as App server 2
  participant n_sessions as Session store
  Note over n_users,n_app: One server does everything
  n_users->>n_app: GET /feed HTTP/1.1
  n_app-->>n_users: HTTP/1.1 200 OK
  Note over n_users,n_app: Traffic doubles, the box is full
  n_users->>n_app: GET /feed HTTP/1.1
  n_app-->>n_users: HTTP/1.1 503 Service Unavailable
  Note over n_users,n_app: One box, one failure
  n_users-xn_app: curl https://api.example.com/feed
  Note over n_lb,n_app2: Horizontal scaling: more boxes
  n_lb->>n_app2: GET /feed HTTP/1.1
  n_app2-->>n_lb: HTTP/1.1 200 OK
  Note over n_lb,n_app2: Users keep getting logged out
  n_lb->>n_app2: GET /cart HTTP/1.1
  n_app2-->>n_lb: HTTP/1.1 302 Found
  Note over n_app2,n_sessions: Make the app servers stateless
  n_app2->>n_sessions: GET session:9c1e7a
  n_sessions-->>n_app2: (bulk string)
  Note over n_lb,n_app2: One server dies, users barely notice
  n_lb-xn_app2: GET /healthz HTTP/1.1

D2

direction: right

n_users: "Users" {
  shape: rectangle
}
n_app: "App server 1" {
  shape: rectangle
}
n_db: "Database" {
  shape: cylinder
}
n_lb: "Load balancer" {
  shape: diamond
}
n_app2: "App server 2" {
  shape: rectangle
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_app3: "App server 3" {
  shape: rectangle
}
n_sessions: "Session store" {
  shape: hexagon
}

n_app -> n_db
n_users -> n_lb
n_lb -> n_users {
  style.stroke-dash: 3
}
n_lb -> n_app
n_lb -> n_app2: "lost" {
  target-arrowhead.shape: cross
}
n_lb -> n_app3
n_app2 -> n_db
n_app3 -> n_db
n_app -> n_sessions
n_app2 -> n_sessions
n_app3 -> n_sessions

PlantUML: request sequence

@startuml
participant "Users" as n_users
participant "App server 1" as n_app
participant "Load balancer" as n_lb
participant "App server 2" as n_app2
collections "Session store" as n_sessions
note over n_users, n_app : One server does everything
n_users -> n_app : GET /feed HTTP/1.1
n_app --> n_users : HTTP/1.1 200 OK
note over n_users, n_app : Traffic doubles, the box is full
n_users -> n_app : GET /feed HTTP/1.1
n_app --> n_users : HTTP/1.1 503 Service Unavailable
note over n_users, n_app : One box, one failure
n_users ->x n_app : [lost] curl https://api.example.com/feed
note over n_lb, n_app2 : Horizontal scaling: more boxes
n_lb -> n_app2 : GET /feed HTTP/1.1
n_app2 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : Users keep getting logged out
n_lb -> n_app2 : GET /cart HTTP/1.1
n_app2 --> n_lb : HTTP/1.1 302 Found
note over n_app2, n_sessions : Make the app servers stateless
n_app2 -> n_sessions : GET session:9c1e7a
n_sessions --> n_app2 : (bulk string)
note over n_lb, n_app2 : One server dies, users barely notice
n_lb ->x n_app2 : [lost] GET /healthz HTTP/1.1
@enduml

PlantUML: components

@startuml
left to right direction
agent "Users" as n_users
component "App server 1" as n_app
database "Database" as n_db
node "Load balancer" as n_lb
component "App server 2" as n_app2 #fde2e2;line:c62828;line.dashed
component "App server 3" as n_app3
stack "Session store" as n_sessions
n_app --> n_db
n_users --> n_lb
n_lb ..> n_users
n_lb --> n_app
n_lb -[#c62828,dashed]-> n_app2 : lost
n_lb --> n_app3
n_app2 --> n_db
n_app3 --> n_db
n_app --> n_sessions
n_app2 --> n_sessions
n_app3 --> n_sessions
@enduml

Recap

Scale up first: it is simple. Scale out when the box gets too expensive, too big, or too risky to be the only one.

1) One server, however big, is a single point of failure. 2) Many servers need a load balancer in front. 3) App servers must be stateless: sessions and files go to shared stores. 4) Keep N+1. 5) The database is still one box: it is the next thing to scale.

Numbers: at the start and at the end

MetricStartEnd
Served300/s2 400/s
Latency p9580 ms240 ms
Errors0%0.5%
Cost per month160 $1 040 $

Steps

  1. Case One server does everything

    A small feed API runs on one 4-vCPU server.

  2. Problem Traffic doubles, the box is full

    After a marketing push, users want 700 requests a second.

  3. Fix Vertical scaling: a bigger box

    The easiest fix: move to a 16-vCPU server.

  4. Problem There is no bigger box

    A year later: 2 400 requests a second.

  5. Problem One box, one failure

    A failed disk takes the server down.

  6. Fix Horizontal scaling: more boxes

    Instead of one 64-vCPU giant, three 8-vCPU servers.

  7. Problem Users keep getting logged out

    Sessions were stored in the memory of the server where the user logged in.

  8. Fix Make the app servers stateless

    Sessions move to a shared, fast session store.

  9. Case One server dies, users barely notice

    app-2 crashes.

  10. Recap Recap

    Scale up first: it is simple.

Terms

Vertical scaling
Replacing a server with a bigger, stronger one. It is simple, but one day there is no bigger box, and it is still a single point of failure.
Horizontal scaling
Adding more servers instead of one bigger one. It needs a load balancer in front and stateless app servers.