Cheat sheet · Reliability
Redundancy and failover
The problem
Every part fails sooner or later. How to find single points of failure, add a spare for each, switch over automatically, and not lose data on the way.
What you learn
- How to spot a single point of failure
- Active-active app servers and health checks
- Active-passive databases: replica, failover, fencing
- What async and sync replication cost, and why zones matter
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_guest(["Guest"])
n_app1["App 1 · zone A"]
n_db1[("DB 1 · zone A")]
n_lb{"Load balancer"}
n_app2["App 2 · zone B"]
n_db2[("DB 2 · zone B")]
n_guest --> n_lb
n_lb --> n_app1
n_lb --x|"lost"| n_app2
n_app2 --x|"lost"| n_db2
n_db2 --x|"lost"| n_db1
n_app1 --> n_db1
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_app2,n_db2 downMermaid: request sequence
sequenceDiagram
participant n_guest as Guest
participant n_app1 as App 1 · zone A
participant n_lb as Load balancer
participant n_app2 as App 2 · zone B
participant n_db1 as DB 1 · zone A
participant n_db2 as DB 2 · zone B
Note over n_guest,n_app1: One server, one point of failure
n_guest-xn_app1: GET /bookings/b_311 HTTP/1.1
Note over n_lb,n_app1: Two app servers behind a load balancer
n_lb->>n_app1: GET /healthz HTTP/1.1
n_app1-->>n_lb: HTTP/1.1 200 OK
Note over n_lb,n_app2: A dead server is taken out in seconds
n_lb-xn_app2: GET /healthz HTTP/1.1
Note over n_db1,n_db2: A standby copy of the database
n_db1->>n_db2: START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2-->>n_db1: streaming WAL to db2
Note over n_app1,n_db1: The primary database dies
n_app1-xn_db1: INSERT INTO bookings (id, room, night)
Note over n_app1,n_db2: Automatic failover
n_app1->>n_db2: resolve db-primary.internal
n_db2-->>n_app1: #59;#59; ANSWER SECTION
Note over n_db1,n_db2: The last seconds never reached the replica
n_db1-xn_db2: last state before db1 crashed
Note over n_app1,n_db1: Two primaries at once
n_app1->>n_db1: INSERT INTO bookings (id, room, night)
n_db1-->>n_app1: INSERT 0 1
Note over n_db2,n_db1: Fence the old primary
n_db2->>n_db1: db1: rejoin as primary, my term=7
n_db1-->>n_db2: rejected: cluster term=8, primary is db2
Note over n_app1,n_db2: Confirm only after the replica has it
n_app1->>n_db2: SET synchronous_commit = on#59;
n_db2-->>n_app1: COMMITD2
direction: right
n_guest: "Guest" {
shape: rectangle
}
n_app1: "App 1 · zone A" {
shape: rectangle
}
n_db1: "DB 1 · zone A" {
shape: cylinder
}
n_lb: "Load balancer" {
shape: diamond
}
n_app2: "App 2 · zone B" {
shape: rectangle
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_db2: "DB 2 · zone B" {
shape: cylinder
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_guest -> n_lb
n_lb -> n_app1
n_lb -> n_app2: "lost" {
target-arrowhead.shape: cross
}
n_app2 -> n_db2: "lost" {
target-arrowhead.shape: cross
}
n_db2 -> n_db1: "lost" {
target-arrowhead.shape: cross
}
n_app1 -> n_db1PlantUML: request sequence
@startuml
participant "Guest" as n_guest
participant "App 1 · zone A" as n_app1
participant "Load balancer" as n_lb
participant "App 2 · zone B" as n_app2
database "DB 1 · zone A" as n_db1
database "DB 2 · zone B" as n_db2
note over n_guest, n_app1 : One server, one point of failure
n_guest ->x n_app1 : [lost] GET /bookings/b_311 HTTP/1.1
note over n_lb, n_app1 : Two app servers behind a load balancer
n_lb -> n_app1 : GET /healthz HTTP/1.1
n_app1 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : A dead server is taken out in seconds
n_lb ->x n_app2 : [lost] GET /healthz HTTP/1.1
note over n_db1, n_db2 : A standby copy of the database
n_db1 -> n_db2 : START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2 --> n_db1 : streaming WAL to db2
note over n_app1, n_db1 : The primary database dies
n_app1 ->x n_db1 : [lost] INSERT INTO bookings (id, room, night)
note over n_app1, n_db2 : Automatic failover
n_app1 -> n_db2 : resolve db-primary.internal
n_db2 --> n_app1 : ;; ANSWER SECTION
note over n_db1, n_db2 : The last seconds never reached the replica
n_db1 ->x n_db2 : [lost] last state before db1 crashed
note over n_app1, n_db1 : Two primaries at once
n_app1 -> n_db1 : [duplicate] INSERT INTO bookings (id, room, night)
n_db1 --> n_app1 : INSERT 0 1
note over n_db2, n_db1 : Fence the old primary
n_db2 -> n_db1 : db1: rejoin as primary, my term=7
n_db1 --> n_db2 : rejected: cluster term=8, primary is db2
note over n_app1, n_db2 : Confirm only after the replica has it
n_app1 -> n_db2 : SET synchronous_commit = on;
n_db2 --> n_app1 : COMMIT
@endumlPlantUML: components
@startuml
left to right direction
agent "Guest" as n_guest
component "App 1 · zone A" as n_app1
database "DB 1 · zone A" as n_db1
node "Load balancer" as n_lb
component "App 2 · zone B" as n_app2 #fde2e2;line:c62828;line.dashed
database "DB 2 · zone B" as n_db2 #fde2e2;line:c62828;line.dashed
n_guest --> n_lb
n_lb --> n_app1
n_lb -[#c62828,dashed]-> n_app2 : lost
n_app2 -[#c62828,dashed]-> n_db2 : lost
n_db2 -[#c62828,dashed]-> n_db1 : lost
n_app1 --> n_db1
@endumlRecap
Find the parts whose loss stops everything and give each a spare that takes over without a human.
1) Stateless servers run active-active behind health checks. 2) Databases run active-passive with a replica and automatic failover. 3) Fence the old primary so only one can write. 4) Async replication can lose the last seconds; sync costs latency. 5) Spread the copies across zones.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Errors | 100% | 0% |
| Time to recover | 1 800 s | 30 s |
| Lost bookings | 0 | 0 |
| Write p95 | 12 ms | 15 ms |
Steps
- Problem One server, one point of failure
The booking site runs on one app server and one database.
- Fix Two app servers behind a load balancer
A second app server runs the same code.
- Case A dead server is taken out in seconds
App 2 hangs.
- Fix A standby copy of the database
A database cannot simply be doubled: two servers taking writes on their own would disagree.
- Problem The primary database dies
DB 1's server loses power.
- Fix Automatic failover
A small cluster manager watches DB 1.
- Problem The last seconds never reached the replica
Replication was asynchronous: DB 1 confirmed a write to the guest before sending it to DB 2.
- Problem Two primaries at once
DB 1 comes back after a reboot.
- Fix Fence the old primary
Every promotion raises a term number.
- Fix Confirm only after the replica has it
Booking writes now use synchronous replication: the primary answers “committed” only after the replica confirms it has the change on disk.
- Case A whole zone goes dark
Zone B loses power: app 2 and DB 2 are gone together.
- Recap Recap
Find the parts whose loss stops everything and give each a spare that takes over without a human.
Terms
- Single point of failure (SPOF)
- A part whose loss stops the whole system. The fix is a spare that takes over without a human.
- Failover
- When the primary database dies, a standby replica is promoted to primary and the app is pointed at it. Automatic failover does this without a human, in seconds.
- Split brain
- Two nodes both believe they are the primary and both accept writes. The data drifts apart, so the old primary must be fenced off.