mufassall

Cheat sheet · Reliability

Redundancy and failover

14 min · Intermediate · mufassall.dtwin.uz/en/l/redundancy/

← Open the lesson

The problem

Every part fails sooner or later. How to find single points of failure, add a spare for each, switch over automatically, and not lose data on the way.

What you learn

  • How to spot a single point of failure
  • Active-active app servers and health checks
  • Active-passive databases: replica, failover, fencing
  • What async and sync replication cost, and why zones matter
Final scene

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_guest(["Guest"])
  n_app1["App 1 · zone A"]
  n_db1[("DB 1 · zone A")]
  n_lb{"Load balancer"}
  n_app2["App 2 · zone B"]
  n_db2[("DB 2 · zone B")]
  n_guest --> n_lb
  n_lb --> n_app1
  n_lb --x|"lost"| n_app2
  n_app2 --x|"lost"| n_db2
  n_db2 --x|"lost"| n_db1
  n_app1 --> n_db1
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_app2,n_db2 down

Mermaid: request sequence

sequenceDiagram
  participant n_guest as Guest
  participant n_app1 as App 1 · zone A
  participant n_lb as Load balancer
  participant n_app2 as App 2 · zone B
  participant n_db1 as DB 1 · zone A
  participant n_db2 as DB 2 · zone B
  Note over n_guest,n_app1: One server, one point of failure
  n_guest-xn_app1: GET /bookings/b_311 HTTP/1.1
  Note over n_lb,n_app1: Two app servers behind a load balancer
  n_lb->>n_app1: GET /healthz HTTP/1.1
  n_app1-->>n_lb: HTTP/1.1 200 OK
  Note over n_lb,n_app2: A dead server is taken out in seconds
  n_lb-xn_app2: GET /healthz HTTP/1.1
  Note over n_db1,n_db2: A standby copy of the database
  n_db1->>n_db2: START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
  n_db2-->>n_db1: streaming WAL to db2
  Note over n_app1,n_db1: The primary database dies
  n_app1-xn_db1: INSERT INTO bookings (id, room, night)
  Note over n_app1,n_db2: Automatic failover
  n_app1->>n_db2: resolve db-primary.internal
  n_db2-->>n_app1: #59;#59; ANSWER SECTION
  Note over n_db1,n_db2: The last seconds never reached the replica
  n_db1-xn_db2: last state before db1 crashed
  Note over n_app1,n_db1: Two primaries at once
  n_app1->>n_db1: INSERT INTO bookings (id, room, night)
  n_db1-->>n_app1: INSERT 0 1
  Note over n_db2,n_db1: Fence the old primary
  n_db2->>n_db1: db1: rejoin as primary, my term=7
  n_db1-->>n_db2: rejected: cluster term=8, primary is db2
  Note over n_app1,n_db2: Confirm only after the replica has it
  n_app1->>n_db2: SET synchronous_commit = on#59;
  n_db2-->>n_app1: COMMIT

D2

direction: right

n_guest: "Guest" {
  shape: rectangle
}
n_app1: "App 1 · zone A" {
  shape: rectangle
}
n_db1: "DB 1 · zone A" {
  shape: cylinder
}
n_lb: "Load balancer" {
  shape: diamond
}
n_app2: "App 2 · zone B" {
  shape: rectangle
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_db2: "DB 2 · zone B" {
  shape: cylinder
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}

n_guest -> n_lb
n_lb -> n_app1
n_lb -> n_app2: "lost" {
  target-arrowhead.shape: cross
}
n_app2 -> n_db2: "lost" {
  target-arrowhead.shape: cross
}
n_db2 -> n_db1: "lost" {
  target-arrowhead.shape: cross
}
n_app1 -> n_db1

PlantUML: request sequence

@startuml
participant "Guest" as n_guest
participant "App 1 · zone A" as n_app1
participant "Load balancer" as n_lb
participant "App 2 · zone B" as n_app2
database "DB 1 · zone A" as n_db1
database "DB 2 · zone B" as n_db2
note over n_guest, n_app1 : One server, one point of failure
n_guest ->x n_app1 : [lost] GET /bookings/b_311 HTTP/1.1
note over n_lb, n_app1 : Two app servers behind a load balancer
n_lb -> n_app1 : GET /healthz HTTP/1.1
n_app1 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : A dead server is taken out in seconds
n_lb ->x n_app2 : [lost] GET /healthz HTTP/1.1
note over n_db1, n_db2 : A standby copy of the database
n_db1 -> n_db2 : START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2 --> n_db1 : streaming WAL to db2
note over n_app1, n_db1 : The primary database dies
n_app1 ->x n_db1 : [lost] INSERT INTO bookings (id, room, night)
note over n_app1, n_db2 : Automatic failover
n_app1 -> n_db2 : resolve db-primary.internal
n_db2 --> n_app1 : ;; ANSWER SECTION
note over n_db1, n_db2 : The last seconds never reached the replica
n_db1 ->x n_db2 : [lost] last state before db1 crashed
note over n_app1, n_db1 : Two primaries at once
n_app1 -> n_db1 : [duplicate] INSERT INTO bookings (id, room, night)
n_db1 --> n_app1 : INSERT 0 1
note over n_db2, n_db1 : Fence the old primary
n_db2 -> n_db1 : db1: rejoin as primary, my term=7
n_db1 --> n_db2 : rejected: cluster term=8, primary is db2
note over n_app1, n_db2 : Confirm only after the replica has it
n_app1 -> n_db2 : SET synchronous_commit = on;
n_db2 --> n_app1 : COMMIT
@enduml

PlantUML: components

@startuml
left to right direction
agent "Guest" as n_guest
component "App 1 · zone A" as n_app1
database "DB 1 · zone A" as n_db1
node "Load balancer" as n_lb
component "App 2 · zone B" as n_app2 #fde2e2;line:c62828;line.dashed
database "DB 2 · zone B" as n_db2 #fde2e2;line:c62828;line.dashed
n_guest --> n_lb
n_lb --> n_app1
n_lb -[#c62828,dashed]-> n_app2 : lost
n_app2 -[#c62828,dashed]-> n_db2 : lost
n_db2 -[#c62828,dashed]-> n_db1 : lost
n_app1 --> n_db1
@enduml

Recap

Find the parts whose loss stops everything and give each a spare that takes over without a human.

1) Stateless servers run active-active behind health checks. 2) Databases run active-passive with a replica and automatic failover. 3) Fence the old primary so only one can write. 4) Async replication can lose the last seconds; sync costs latency. 5) Spread the copies across zones.

Numbers: at the start and at the end

MetricStartEnd
Errors100%0%
Time to recover1 800 s30 s
Lost bookings00
Write p9512 ms15 ms

Steps

  1. Problem One server, one point of failure

    The booking site runs on one app server and one database.

  2. Fix Two app servers behind a load balancer

    A second app server runs the same code.

  3. Case A dead server is taken out in seconds

    App 2 hangs.

  4. Fix A standby copy of the database

    A database cannot simply be doubled: two servers taking writes on their own would disagree.

  5. Problem The primary database dies

    DB 1's server loses power.

  6. Fix Automatic failover

    A small cluster manager watches DB 1.

  7. Problem The last seconds never reached the replica

    Replication was asynchronous: DB 1 confirmed a write to the guest before sending it to DB 2.

  8. Problem Two primaries at once

    DB 1 comes back after a reboot.

  9. Fix Fence the old primary

    Every promotion raises a term number.

  10. Fix Confirm only after the replica has it

    Booking writes now use synchronous replication: the primary answers “committed” only after the replica confirms it has the change on disk.

  11. Case A whole zone goes dark

    Zone B loses power: app 2 and DB 2 are gone together.

  12. Recap Recap

    Find the parts whose loss stops everything and give each a spare that takes over without a human.

Terms

Single point of failure (SPOF)
A part whose loss stops the whole system. The fix is a spare that takes over without a human.
Failover
When the primary database dies, a standby replica is promoted to primary and the app is pointed at it. Automatic failover does this without a human, in seconds.
Split brain
Two nodes both believe they are the primary and both accept writes. The data drifts apart, so the old primary must be fenced off.