Konspekt · Ma'lumot va saqlash
Konsensus va lider saylovi
Muammo
Tungi to'lov job'ining ikki nusxasi uni qaysi biri bajarishini kelishib olishi kerak. Bir-birini ping qilish ishlamaydi, bitta lock server yagona nosozlik nuqtasi. Besh node'li klaster qanday qilib lider saylaydi, replikatsiya qilingan log yuritadi va ko'pchilikni sanab split brain'dan qochadi.
Nimani o'rganasiz
- Nega ikki node kim lider ekanini o'zlari xavfsiz hal qila olmaydi
- Raft uslubidagi klaster log'ni qanday ko'chiradi va ko'pchilikda commit qiladi
- Lider o'lsa yoki uzilsa term va ovozlar nima qiladi
- Nega lock'ga fencing token kerak va uni qayerda saqlash kerak (etcd, ZooKeeper)
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_w1("To'lov worker 1")
n_w2("To'lov worker 2")
n_ledger[("To'lovlar bazasi")]
n_n1["Node 1"]
n_n2["Node 2"]
n_n3["Node 3"]
n_n4["Node 4"]
n_n5["Node 5"]
n_w1 --x n_ledger
n_w2 --> n_ledger
n_n2 --> n_n3
n_n2 --> n_n4
n_w1 --> n_n2
n_n2 --> n_n1
n_n2 --> n_n5
classDef slow fill:#fff4d6,stroke:#b26a00,color:#3d2600
class n_w1 slowMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_w2 as To'lov worker 2
participant n_ledger as To'lovlar bazasi
participant n_n1 as Node 1
participant n_w1 as To'lov worker 1
participant n_n2 as Node 2
participant n_n3 as Node 3
Note over n_w2,n_ledger: Ikki worker, ikkalasi ham o'zini boshliq deb biladi
n_w2->>n_ledger: INSERT INTO payouts (run, merchant, amount)
n_ledger-->>n_w2: INSERT 0 312
Note over n_w2,n_n1: Uchinchi tomondan so'ra: bitta lock server
n_w2->>n_n1: POST /v1/locks/payouts HTTP/1.1
n_n1-->>n_w2: HTTP/1.1 409 Conflict
Note over n_w1,n_n1: Lock server — yangi yagona nosozlik nuqtasi
n_w1-xn_n1: POST /v1/locks/payouts HTTP/1.1
Note over n_n1,n_n2: Besh node, bitta lider, bitta ko'chiriladigan log
n_n1->>n_n2: AppendEntries n1 -#62; n2
n_n2-->>n_n1: AppendEntries reply
Note over n_w1,n_n1: Yozuv ko'pchilikda bo'lsa, xavfsiz
n_w1->>n_n1: POST /v1/locks/payouts HTTP/1.1
n_n1-->>n_w1: HTTP/1.1 200 OK
Note over n_w1,n_n1: Lider bitta follower bilan uzilib qoldi
n_w1->>n_n1: POST /v1/locks/payouts HTTP/1.1
n_n1-->>n_w1: HTTP/1.1 503 Service Unavailable
Note over n_n2,n_n3: Ko'pchilik tomon yangi lider saylaydi
n_n2->>n_n3: RequestVote n2 -#62; n3
n_n3-->>n_n2: RequestVote reply
Note over n_n2,n_n1: Eski lider qaytadi va o'rnini bo'shatadi
n_n2->>n_n1: INFO n1: AppendEntries with term 2 #62; my term 1
Note over n_w1,n_ledger: To'xtab qolgan worker eski lock bilan uyg'onadi
n_w1->>n_ledger: UPDATE payout_runs SET fence = 42, status = 'running'
n_ledger-->>n_w1: UPDATE 0D2
direction: right
n_w1: "To'lov worker 1" {
shape: rectangle
style.multiple: true
style.fill: "#fff4d6"
style.stroke: "#b26a00"
}
n_w2: "To'lov worker 2" {
shape: rectangle
style.multiple: true
}
n_ledger: "To'lovlar bazasi" {
shape: cylinder
}
n_n1: "Node 1" {
shape: rectangle
}
n_n2: "Node 2" {
shape: rectangle
}
n_n3: "Node 3" {
shape: rectangle
}
n_n4: "Node 4" {
shape: rectangle
}
n_n5: "Node 5" {
shape: rectangle
}
n_w1 -> n_ledger {
target-arrowhead.shape: cross
}
n_w2 -> n_ledger
n_n2 -> n_n3
n_n2 -> n_n4
n_w1 -> n_n2
n_n2 -> n_n1
n_n2 -> n_n5PlantUML: so'rovlar ketma-ketligi
@startuml
participant "To'lov worker 2" as n_w2
database "To'lovlar bazasi" as n_ledger
participant "Node 1" as n_n1
participant "To'lov worker 1" as n_w1
participant "Node 2" as n_n2
participant "Node 3" as n_n3
note over n_w2, n_ledger : Ikki worker, ikkalasi ham o'zini boshliq deb biladi
n_w2 -> n_ledger : [dublikat] INSERT INTO payouts (run, merchant, amount)
n_ledger --> n_w2 : INSERT 0 312
note over n_w2, n_n1 : Uchinchi tomondan so'ra: bitta lock server
n_w2 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w2 : HTTP/1.1 409 Conflict
note over n_w1, n_n1 : Lock server — yangi yagona nosozlik nuqtasi
n_w1 ->x n_n1 : [yo'qoldi] POST /v1/locks/payouts HTTP/1.1
note over n_n1, n_n2 : Besh node, bitta lider, bitta ko'chiriladigan log
n_n1 -> n_n2 : AppendEntries n1 -> n2
n_n2 --> n_n1 : AppendEntries reply
note over n_w1, n_n1 : Yozuv ko'pchilikda bo'lsa, xavfsiz
n_w1 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w1 : HTTP/1.1 200 OK
note over n_w1, n_n1 : Lider bitta follower bilan uzilib qoldi
n_w1 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w1 : HTTP/1.1 503 Service Unavailable
note over n_n2, n_n3 : Ko'pchilik tomon yangi lider saylaydi
n_n2 -> n_n3 : RequestVote n2 -> n3
n_n3 --> n_n2 : RequestVote reply
note over n_n2, n_n1 : Eski lider qaytadi va o'rnini bo'shatadi
n_n2 -> n_n1 : INFO n1: AppendEntries with term 2 > my term 1
note over n_w1, n_ledger : To'xtab qolgan worker eski lock bilan uyg'onadi
n_w1 -> n_ledger : UPDATE payout_runs SET fence = 42, status = 'running'
n_ledger --> n_w1 : UPDATE 0
@endumlPlantUML: komponentlar
@startuml
left to right direction
collections "To'lov worker 1" as n_w1 #fff4d6;line:b26a00
collections "To'lov worker 2" as n_w2
database "To'lovlar bazasi" as n_ledger
component "Node 1" as n_n1
component "Node 2" as n_n2
component "Node 3" as n_n3
component "Node 4" as n_n4
component "Node 5" as n_n5
n_w1 -[#c62828]-> n_ledger
n_w2 --> n_ledger
n_n2 --> n_n3
n_n2 --> n_n4
n_w1 --> n_n2
n_n2 --> n_n1
n_n2 --> n_n5
@endumlXulosa
Lider bo'yicha kelishish uchun ping emas, ko'pchilik kerak.
1) 3 yoki 5 node ishlating: ular 1 yoki 2 nosozlikka chidaydi. 2) O'zgarish faqat ko'pchilik saqlaganda commit bo'ladi. 3) Term va ovozlar har term'da ko'pi bilan bitta lider beradi; ozchilik hech qachon commit qila olmaydi. 4) Lock fencing token olib yuradi, himoyalangan resurs uni tekshiradi. Buni o'zingiz yozmang: lock, lider va config'ni etcd, ZooKeeper yoki Consul'da saqlang.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Ikki marta to'langan | 48 000 $ | 0 $ |
| Lock yo'q vaqt | 0 s | 0,4 s |
| Lock yozish vaqti | 2 ms | 7 ms |
Qadamlar
- Muammo Ikki worker, ikkalasi ham o'zini boshliq deb biladi
Tungi to'lov job'i ishonchlilik uchun ikki serverda turadi.
- Yechim Uchinchi tomondan so'ra: bitta lock server
Endi qarorni alohida lock server qabul qiladi.
- Muammo Lock server — yangi yagona nosozlik nuqtasi
Lock serverning diski ishdan chiqdi.
- Yechim Besh node, bitta lider, bitta ko'chiriladigan log
Endi lock servisi besh node'da ishlaydi.
- Holat Yozuv ko'pchilikda bo'lsa, xavfsiz
Worker 1 lock so'raydi.
- Muammo Lider bitta follower bilan uzilib qoldi
Switch ishdan chiqdi.
- Yechim Ko'pchilik tomon yangi lider saylaydi
Node 2, 3 va 4 liderdan xabar olmay qoladi.
- Holat Eski lider qaytadi va o'rnini bo'shatadi
Switch almashtirildi.
- Holat To'xtab qolgan worker eski lock bilan uyg'onadi
Worker 1 uzoq garbage collection pauzasida 20 s qotib qoladi.
- Xulosa Xulosa
Lider bo'yicha kelishish uchun ping emas, ko'pchilik kerak.
Atamalar
- Konsensus (Raft)
- Bir nechta node lider va umumiy log bo'yicha ko'pchilik ovozi bilan kelishadi. O'zgarish ko'pchilik saqlaganda commit bo'ladi, term va ovozlar har term'da ko'pi bilan bitta lider beradi.
- Fencing token
- Har yangi lock bilan beriladigan o'suvchi raqam. Himoyalangan resurs ko'rgan eng katta raqamni eslab qoladi va eski lock bilan uyg'ongan worker'ning yozuvini rad etadi.