Konspekt · Ishonchlilik
Zaxira va failover
Muammo
Har qanday qism ertami-kechmi ishdan chiqadi. Yagona nosozlik nuqtalarini topish, har biriga zaxira qo'yish, avtomatik almashtirish va bunda ma'lumotni yo'qotmaslik.
Nimani o'rganasiz
- Yagona nosozlik nuqtasini (single point of failure) qanday topish
- Active-active app serverlar va health check
- Active-passive baza: replika, failover, fencing
- Asinxron va sinxron replikatsiya narxi va nega zonalar muhim
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_guest(["Mehmon"])
n_app1["App 1 · A zona"]
n_db1[("Baza 1 · A zona")]
n_lb{"Load balancer"}
n_app2["App 2 · B zona"]
n_db2[("Baza 2 · B zona")]
n_guest --> n_lb
n_lb --> n_app1
n_lb --x|"yo'qoldi"| n_app2
n_app2 --x|"yo'qoldi"| n_db2
n_db2 --x|"yo'qoldi"| n_db1
n_app1 --> n_db1
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_app2,n_db2 downMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_guest as Mehmon
participant n_app1 as App 1 · A zona
participant n_lb as Load balancer
participant n_app2 as App 2 · B zona
participant n_db1 as Baza 1 · A zona
participant n_db2 as Baza 2 · B zona
Note over n_guest,n_app1: Bitta server — bitta nosozlik nuqtasi
n_guest-xn_app1: GET /bookings/b_311 HTTP/1.1
Note over n_lb,n_app1: Load balancer ortida ikkita app server
n_lb->>n_app1: GET /healthz HTTP/1.1
n_app1-->>n_lb: HTTP/1.1 200 OK
Note over n_lb,n_app2: O'lik server soniyalarda chiqariladi
n_lb-xn_app2: GET /healthz HTTP/1.1
Note over n_db1,n_db2: Bazaning zaxira nusxasi
n_db1->>n_db2: START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2-->>n_db1: streaming WAL to db2
Note over n_app1,n_db1: Asosiy baza o'ladi
n_app1-xn_db1: INSERT INTO bookings (id, room, night)
Note over n_app1,n_db2: Avtomatik failover
n_app1->>n_db2: resolve db-primary.internal
n_db2-->>n_app1: #59;#59; ANSWER SECTION
Note over n_db1,n_db2: Oxirgi soniyalar replikaga yetmadi
n_db1-xn_db2: last state before db1 crashed
Note over n_app1,n_db1: Bir vaqtda ikki primary
n_app1->>n_db1: INSERT INTO bookings (id, room, night)
n_db1-->>n_app1: INSERT 0 1
Note over n_db2,n_db1: Eski primary'ni o'rab qo'yish
n_db2->>n_db1: db1: rejoin as primary, my term=7
n_db1-->>n_db2: rejected: cluster term=8, primary is db2
Note over n_app1,n_db2: Replika olgandan keyingina tasdiqlash
n_app1->>n_db2: SET synchronous_commit = on#59;
n_db2-->>n_app1: COMMITD2
direction: right
n_guest: "Mehmon" {
shape: rectangle
}
n_app1: "App 1 · A zona" {
shape: rectangle
}
n_db1: "Baza 1 · A zona" {
shape: cylinder
}
n_lb: "Load balancer" {
shape: diamond
}
n_app2: "App 2 · B zona" {
shape: rectangle
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_db2: "Baza 2 · B zona" {
shape: cylinder
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_guest -> n_lb
n_lb -> n_app1
n_lb -> n_app2: "yo'qoldi" {
target-arrowhead.shape: cross
}
n_app2 -> n_db2: "yo'qoldi" {
target-arrowhead.shape: cross
}
n_db2 -> n_db1: "yo'qoldi" {
target-arrowhead.shape: cross
}
n_app1 -> n_db1PlantUML: so'rovlar ketma-ketligi
@startuml
participant "Mehmon" as n_guest
participant "App 1 · A zona" as n_app1
participant "Load balancer" as n_lb
participant "App 2 · B zona" as n_app2
database "Baza 1 · A zona" as n_db1
database "Baza 2 · B zona" as n_db2
note over n_guest, n_app1 : Bitta server — bitta nosozlik nuqtasi
n_guest ->x n_app1 : [yo'qoldi] GET /bookings/b_311 HTTP/1.1
note over n_lb, n_app1 : Load balancer ortida ikkita app server
n_lb -> n_app1 : GET /healthz HTTP/1.1
n_app1 --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_app2 : O'lik server soniyalarda chiqariladi
n_lb ->x n_app2 : [yo'qoldi] GET /healthz HTTP/1.1
note over n_db1, n_db2 : Bazaning zaxira nusxasi
n_db1 -> n_db2 : START_REPLICATION SLOT db2 PHYSICAL 0/5A3F1C0
n_db2 --> n_db1 : streaming WAL to db2
note over n_app1, n_db1 : Asosiy baza o'ladi
n_app1 ->x n_db1 : [yo'qoldi] INSERT INTO bookings (id, room, night)
note over n_app1, n_db2 : Avtomatik failover
n_app1 -> n_db2 : resolve db-primary.internal
n_db2 --> n_app1 : ;; ANSWER SECTION
note over n_db1, n_db2 : Oxirgi soniyalar replikaga yetmadi
n_db1 ->x n_db2 : [yo'qoldi] last state before db1 crashed
note over n_app1, n_db1 : Bir vaqtda ikki primary
n_app1 -> n_db1 : [dublikat] INSERT INTO bookings (id, room, night)
n_db1 --> n_app1 : INSERT 0 1
note over n_db2, n_db1 : Eski primary'ni o'rab qo'yish
n_db2 -> n_db1 : db1: rejoin as primary, my term=7
n_db1 --> n_db2 : rejected: cluster term=8, primary is db2
note over n_app1, n_db2 : Replika olgandan keyingina tasdiqlash
n_app1 -> n_db2 : SET synchronous_commit = on;
n_db2 --> n_app1 : COMMIT
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Mehmon" as n_guest
component "App 1 · A zona" as n_app1
database "Baza 1 · A zona" as n_db1
node "Load balancer" as n_lb
component "App 2 · B zona" as n_app2 #fde2e2;line:c62828;line.dashed
database "Baza 2 · B zona" as n_db2 #fde2e2;line:c62828;line.dashed
n_guest --> n_lb
n_lb --> n_app1
n_lb -[#c62828,dashed]-> n_app2 : yo'qoldi
n_app2 -[#c62828,dashed]-> n_db2 : yo'qoldi
n_db2 -[#c62828,dashed]-> n_db1 : yo'qoldi
n_app1 --> n_db1
@endumlXulosa
Yo'qolsa hammasini to'xtatadigan qismlarni toping va har biriga odamsiz o'rnini egallaydigan zaxira bering.
1) Stateless serverlar health check ortida active-active ishlaydi. 2) Baza replika va avtomatik failover bilan active-passive. 3) Eski primary'ni fencing qiling, yozadigani bitta bo'lsin. 4) Asinxron replikatsiya oxirgi soniyalarni yo'qotishi mumkin; sinxron kechikish qo'shadi. 5) Nusxalarni zonalarga tarqating.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Xatolar | 100% | 0% |
| Tiklanish vaqti | 1 800 s | 30 s |
| Yo'qolgan bron | 0 | 0 |
| Yozish p95 | 12 ms | 15 ms |
Qadamlar
- Muammo Bitta server — bitta nosozlik nuqtasi
Mehmonxona bron sayti bitta app server va bitta bazada ishlaydi.
- Yechim Load balancer ortida ikkita app server
Ikkinchi app server o'sha kodni ishlatadi.
- Holat O'lik server soniyalarda chiqariladi
App 2 qotib qoldi.
- Yechim Bazaning zaxira nusxasi
Bazani shunchaki ikkilantirib bo'lmaydi: ikkita server o'zicha yozuv qabul qilsa, ular kelisha olmaydi.
- Muammo Asosiy baza o'ladi
Baza 1 serverining toki o'chadi.
- Yechim Avtomatik failover
Kichik klaster menejeri Baza 1 ni kuzatadi.
- Muammo Oxirgi soniyalar replikaga yetmadi
Replikatsiya asinxron edi: Baza 1 yozuvni Baza 2 ga yuborishdan oldin mehmonga tasdiqlab qo'ygan.
- Muammo Bir vaqtda ikki primary
Baza 1 qayta yuklangach qaytadi.
- Yechim Eski primary'ni o'rab qo'yish
Har promote term raqamini oshiradi.
- Yechim Replika olgandan keyingina tasdiqlash
Bron yozuvlari endi sinxron replikatsiya ishlatadi: primary «commit bo'ldi» deb faqat replika o'zgarishni diskka yozganini tasdiqlagandan keyin javob beradi.
- Holat Butun zona o'chadi
B zonada tok o'chadi: app 2 va Baza 2 birga yo'qoladi.
- Xulosa Xulosa
Yo'qolsa hammasini to'xtatadigan qismlarni toping va har biriga odamsiz o'rnini egallaydigan zaxira bering.
Atamalar
- Yagona nosozlik nuqtasi (SPOF)
- Yo'qolsa butun tizimni to'xtatadigan qism. Unga odamsiz o'rnini egallaydigan zaxira beriladi.
- Failover
- Asosiy baza o'lganda zaxira replikani primary qilib ko'tarish va ilovani unga yo'naltirish. Avtomatik failover buni odamsiz, soniyalar ichida qiladi.
- Split brain
- Ikki node bir vaqtda o'zini primary deb hisoblab, ikkalasi ham yozishi. Ma'lumot ikki tomonga ajraladi, shuning uchun eski primary fencing bilan o'rab qo'yiladi.