mufassall

Konspekt · Ishonchlilik

Zaxira va failover

14 daq · O'rta · mufassall.dtwin.uz/uz/l/redundancy/

← Darsni ochish

Muammo

Har qanday qism ertami-kechmi ishdan chiqadi. Yagona nosozlik nuqtalarini topish, har biriga zaxira qo'yish, avtomatik almashtirish va bunda ma'lumotni yo'qotmaslik.

Nimani o'rganasiz

  • Yagona nosozlik nuqtasini (single point of failure) qanday topish
  • Active-active app serverlar va health check
  • Active-passive baza: replika, failover, fencing
  • Asinxron va sinxron replikatsiya narxi va nega zonalar muhim
Yakuniy sahna

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_guest(["Mehmon"])
  n_app1["App 1 · A zona"]
  n_db1[("Baza 1 · A zona")]
  n_lb{"Load balancer"}
  n_app2["App 2 · B zona"]
  n_db2[("Baza 2 · B zona")]
  n_guest --> n_lb
  n_lb --> n_app1
  n_lb --x|"yo'qoldi"| n_app2
  n_app2 --x|"yo'qoldi"| n_db2
  n_db2 --x|"yo'qoldi"| n_db1
  n_app1 --> n_db1
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_app2,n_db2 down

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_guest as Mehmon
  participant n_app1 as App 1 · A zona
  participant n_lb as Load balancer
  participant n_app2 as App 2 · B zona
  participant n_db1 as Baza 1 · A zona
  participant n_db2 as Baza 2 · B zona
  Note over n_guest,n_app1: Bitta server — bitta nosozlik nuqtasi
  n_guest-xn_app1: GET /bookings/b_311 HTTP/1.1
  Note over n_lb,n_app1: Load balancer ortida ikkita app server
  n_lb->>n_app1: GET /healthz HTTP/1.1
  n_app1-->>n_lb: HTTP/1.1 200 OK
  Note over n_lb,n_app2: O'lik server soniyalarda chiqariladi
  n_lb-xn_app2: GET /healthz HTTP/1.1
  Note over n_db1,n_db2: Bazaning zaxira nusxasi
  n_db1->>n_db2: START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
  n_db2-->>n_db1: streaming WAL to db2
  Note over n_app1,n_db1: Asosiy baza o'ladi
  n_app1-xn_db1: INSERT INTO bookings (id, room, night)
  Note over n_app1,n_db2: Avtomatik failover
  n_app1->>n_db2: resolve db-primary.internal
  n_db2-->>n_app1: #59;#59; ANSWER SECTION
  Note over n_db1,n_db2: Oxirgi soniyalar replikaga yetmadi
  n_db1-xn_db2: last state before db1 crashed
  Note over n_app1,n_db1: Bir vaqtda ikki primary
  n_app1->>n_db1: INSERT INTO bookings (id, room, night)
  n_db1-->>n_app1: INSERT 0 1
  Note over n_db2,n_db1: Eski primary'ni o'rab qo'yish
  n_db2->>n_db1: db1: rejoin as primary, my term=7
  n_db1-->>n_db2: rejected: cluster term=8, primary is db2
  Note over n_app1,n_db2: Replika olgandan keyingina tasdiqlash
  n_app1->>n_db2: SET synchronous_commit = on#59;
  n_db2-->>n_app1: COMMIT

D2

direction: right

n_guest: "Mehmon" {
  shape: rectangle
}
n_app1: "App 1 · A zona" {
  shape: rectangle
}
n_db1: "Baza 1 · A zona" {
  shape: cylinder
}
n_lb: "Load balancer" {
  shape: diamond
}
n_app2: "App 2 · B zona" {
  shape: rectangle
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_db2: "Baza 2 · B zona" {
  shape: cylinder
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}

n_guest -> n_lb
n_lb -> n_app1
n_lb -> n_app2: "yo'qoldi" {
  target-arrowhead.shape: cross
}
n_app2 -> n_db2: "yo'qoldi" {
  target-arrowhead.shape: cross
}
n_db2 -> n_db1: "yo'qoldi" {
  target-arrowhead.shape: cross
}
n_app1 -> n_db1

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Mehmon" as n_guest
participant "App 1 · A zona" as n_app1
participant "Load balancer" as n_lb
participant "App 2 · B zona" as n_app2
database "Baza 1 · A zona" as n_db1
database "Baza 2 · B zona" as n_db2
note over n_guest, n_app1 : Bitta server — bitta nosozlik nuqtasi
n_guest ->x n_app1 : [yo'qoldi] GET /bookings/b_311 HTTP/1.1
note over n_lb, n_app1 : Load balancer ortida ikkita app server
n_lb -> n_app1 : GET /healthz HTTP/1.1
n_app1 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : O'lik server soniyalarda chiqariladi
n_lb ->x n_app2 : [yo'qoldi] GET /healthz HTTP/1.1
note over n_db1, n_db2 : Bazaning zaxira nusxasi
n_db1 -> n_db2 : START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2 --> n_db1 : streaming WAL to db2
note over n_app1, n_db1 : Asosiy baza o'ladi
n_app1 ->x n_db1 : [yo'qoldi] INSERT INTO bookings (id, room, night)
note over n_app1, n_db2 : Avtomatik failover
n_app1 -> n_db2 : resolve db-primary.internal
n_db2 --> n_app1 : ;; ANSWER SECTION
note over n_db1, n_db2 : Oxirgi soniyalar replikaga yetmadi
n_db1 ->x n_db2 : [yo'qoldi] last state before db1 crashed
note over n_app1, n_db1 : Bir vaqtda ikki primary
n_app1 -> n_db1 : [dublikat] INSERT INTO bookings (id, room, night)
n_db1 --> n_app1 : INSERT 0 1
note over n_db2, n_db1 : Eski primary'ni o'rab qo'yish
n_db2 -> n_db1 : db1: rejoin as primary, my term=7
n_db1 --> n_db2 : rejected: cluster term=8, primary is db2
note over n_app1, n_db2 : Replika olgandan keyingina tasdiqlash
n_app1 -> n_db2 : SET synchronous_commit = on;
n_db2 --> n_app1 : COMMIT
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Mehmon" as n_guest
component "App 1 · A zona" as n_app1
database "Baza 1 · A zona" as n_db1
node "Load balancer" as n_lb
component "App 2 · B zona" as n_app2 #fde2e2;line:c62828;line.dashed
database "Baza 2 · B zona" as n_db2 #fde2e2;line:c62828;line.dashed
n_guest --> n_lb
n_lb --> n_app1
n_lb -[#c62828,dashed]-> n_app2 : yo'qoldi
n_app2 -[#c62828,dashed]-> n_db2 : yo'qoldi
n_db2 -[#c62828,dashed]-> n_db1 : yo'qoldi
n_app1 --> n_db1
@enduml

Xulosa

Yo'qolsa hammasini to'xtatadigan qismlarni toping va har biriga odamsiz o'rnini egallaydigan zaxira bering.

1) Stateless serverlar health check ortida active-active ishlaydi. 2) Baza replika va avtomatik failover bilan active-passive. 3) Eski primary'ni fencing qiling, yozadigani bitta bo'lsin. 4) Asinxron replikatsiya oxirgi soniyalarni yo'qotishi mumkin; sinxron kechikish qo'shadi. 5) Nusxalarni zonalarga tarqating.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Xatolar100%0%
Tiklanish vaqti1 800 s30 s
Yo'qolgan bron00
Yozish p9512 ms15 ms

Qadamlar

  1. Muammo Bitta server — bitta nosozlik nuqtasi

    Mehmonxona bron sayti bitta app server va bitta bazada ishlaydi.

  2. Yechim Load balancer ortida ikkita app server

    Ikkinchi app server o'sha kodni ishlatadi.

  3. Holat O'lik server soniyalarda chiqariladi

    App 2 qotib qoldi.

  4. Yechim Bazaning zaxira nusxasi

    Bazani shunchaki ikkilantirib bo'lmaydi: ikkita server o'zicha yozuv qabul qilsa, ular kelisha olmaydi.

  5. Muammo Asosiy baza o'ladi

    Baza 1 serverining toki o'chadi.

  6. Yechim Avtomatik failover

    Kichik klaster menejeri Baza 1 ni kuzatadi.

  7. Muammo Oxirgi soniyalar replikaga yetmadi

    Replikatsiya asinxron edi: Baza 1 yozuvni Baza 2 ga yuborishdan oldin mehmonga tasdiqlab qo'ygan.

  8. Muammo Bir vaqtda ikki primary

    Baza 1 qayta yuklangach qaytadi.

  9. Yechim Eski primary'ni o'rab qo'yish

    Har promote term raqamini oshiradi.

  10. Yechim Replika olgandan keyingina tasdiqlash

    Bron yozuvlari endi sinxron replikatsiya ishlatadi: primary «commit bo'ldi» deb faqat replika o'zgarishni diskka yozganini tasdiqlagandan keyin javob beradi.

  11. Holat Butun zona o'chadi

    B zonada tok o'chadi: app 2 va Baza 2 birga yo'qoladi.

  12. Xulosa Xulosa

    Yo'qolsa hammasini to'xtatadigan qismlarni toping va har biriga odamsiz o'rnini egallaydigan zaxira bering.

Atamalar

Yagona nosozlik nuqtasi (SPOF)
Yo'qolsa butun tizimni to'xtatadigan qism. Unga odamsiz o'rnini egallaydigan zaxira beriladi.
Failover
Asosiy baza o'lganda zaxira replikani primary qilib ko'tarish va ilovani unga yo'naltirish. Avtomatik failover buni odamsiz, soniyalar ichida qiladi.
Split brain
Ikki node bir vaqtda o'zini primary deb hisoblab, ikkalasi ham yozishi. Ma'lumot ikki tomonga ajraladi, shuning uchun eski primary fencing bilan o'rab qo'yiladi.