mufassall

Konspekt · Ma'lumot va saqlash

Konsensus va lider saylovi

15 daq · Yuqori · mufassall.dtwin.uz/uz/l/consensus/

← Darsni ochish

Muammo

Tungi to'lov job'ining ikki nusxasi uni qaysi biri bajarishini kelishib olishi kerak. Bir-birini ping qilish ishlamaydi, bitta lock server yagona nosozlik nuqtasi. Besh node'li klaster qanday qilib lider saylaydi, replikatsiya qilingan log yuritadi va ko'pchilikni sanab split brain'dan qochadi.

Nimani o'rganasiz

  • Nega ikki node kim lider ekanini o'zlari xavfsiz hal qila olmaydi
  • Raft uslubidagi klaster log'ni qanday ko'chiradi va ko'pchilikda commit qiladi
  • Lider o'lsa yoki uzilsa term va ovozlar nima qiladi
  • Nega lock'ga fencing token kerak va uni qayerda saqlash kerak (etcd, ZooKeeper)
Yakuniy sahna
30%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_w1("To'lov worker 1")
  n_w2("To'lov worker 2")
  n_ledger[("To'lovlar bazasi")]
  n_n1["Node 1"]
  n_n2["Node 2"]
  n_n3["Node 3"]
  n_n4["Node 4"]
  n_n5["Node 5"]
  n_w1 --x n_ledger
  n_w2 --> n_ledger
  n_n2 --> n_n3
  n_n2 --> n_n4
  n_w1 --> n_n2
  n_n2 --> n_n1
  n_n2 --> n_n5
  classDef slow fill:#fff4d6,stroke:#b26a00,color:#3d2600
  class n_w1 slow

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_w2 as To'lov worker 2
  participant n_ledger as To'lovlar bazasi
  participant n_n1 as Node 1
  participant n_w1 as To'lov worker 1
  participant n_n2 as Node 2
  participant n_n3 as Node 3
  Note over n_w2,n_ledger: Ikki worker, ikkalasi ham o'zini boshliq deb biladi
  n_w2->>n_ledger: INSERT INTO payouts (run, merchant, amount)
  n_ledger-->>n_w2: INSERT 0 312
  Note over n_w2,n_n1: Uchinchi tomondan so'ra: bitta lock server
  n_w2->>n_n1: POST /v1/locks/payouts HTTP/1.1
  n_n1-->>n_w2: HTTP/1.1 409 Conflict
  Note over n_w1,n_n1: Lock server — yangi yagona nosozlik nuqtasi
  n_w1-xn_n1: POST /v1/locks/payouts HTTP/1.1
  Note over n_n1,n_n2: Besh node, bitta lider, bitta ko'chiriladigan log
  n_n1->>n_n2: AppendEntries n1 -#62; n2
  n_n2-->>n_n1: AppendEntries reply
  Note over n_w1,n_n1: Yozuv ko'pchilikda bo'lsa, xavfsiz
  n_w1->>n_n1: POST /v1/locks/payouts HTTP/1.1
  n_n1-->>n_w1: HTTP/1.1 200 OK
  Note over n_w1,n_n1: Lider bitta follower bilan uzilib qoldi
  n_w1->>n_n1: POST /v1/locks/payouts HTTP/1.1
  n_n1-->>n_w1: HTTP/1.1 503 Service Unavailable
  Note over n_n2,n_n3: Ko'pchilik tomon yangi lider saylaydi
  n_n2->>n_n3: RequestVote n2 -#62; n3
  n_n3-->>n_n2: RequestVote reply
  Note over n_n2,n_n1: Eski lider qaytadi va o'rnini bo'shatadi
  n_n2->>n_n1: INFO n1: AppendEntries with term 2 #62; my term 1
  Note over n_w1,n_ledger: To'xtab qolgan worker eski lock bilan uyg'onadi
  n_w1->>n_ledger: UPDATE payout_runs SET fence = 42, status = 'running'
  n_ledger-->>n_w1: UPDATE 0

D2

direction: right

n_w1: "To'lov worker 1" {
  shape: rectangle
  style.multiple: true
  style.fill: "#fff4d6"
  style.stroke: "#b26a00"
}
n_w2: "To'lov worker 2" {
  shape: rectangle
  style.multiple: true
}
n_ledger: "To'lovlar bazasi" {
  shape: cylinder
}
n_n1: "Node 1" {
  shape: rectangle
}
n_n2: "Node 2" {
  shape: rectangle
}
n_n3: "Node 3" {
  shape: rectangle
}
n_n4: "Node 4" {
  shape: rectangle
}
n_n5: "Node 5" {
  shape: rectangle
}

n_w1 -> n_ledger {
  target-arrowhead.shape: cross
}
n_w2 -> n_ledger
n_n2 -> n_n3
n_n2 -> n_n4
n_w1 -> n_n2
n_n2 -> n_n1
n_n2 -> n_n5

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "To'lov worker 2" as n_w2
database "To'lovlar bazasi" as n_ledger
participant "Node 1" as n_n1
participant "To'lov worker 1" as n_w1
participant "Node 2" as n_n2
participant "Node 3" as n_n3
note over n_w2, n_ledger : Ikki worker, ikkalasi ham o'zini boshliq deb biladi
n_w2 -> n_ledger : [dublikat] INSERT INTO payouts (run, merchant, amount)
n_ledger --> n_w2 : INSERT 0 312
note over n_w2, n_n1 : Uchinchi tomondan so'ra: bitta lock server
n_w2 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w2 : HTTP/1.1 409 Conflict
note over n_w1, n_n1 : Lock server — yangi yagona nosozlik nuqtasi
n_w1 ->x n_n1 : [yo'qoldi] POST /v1/locks/payouts HTTP/1.1
note over n_n1, n_n2 : Besh node, bitta lider, bitta ko'chiriladigan log
n_n1 -> n_n2 : AppendEntries n1 -> n2
n_n2 --> n_n1 : AppendEntries reply
note over n_w1, n_n1 : Yozuv ko'pchilikda bo'lsa, xavfsiz
n_w1 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w1 : HTTP/1.1 200 OK
note over n_w1, n_n1 : Lider bitta follower bilan uzilib qoldi
n_w1 -> n_n1 : POST /v1/locks/payouts HTTP/1.1
n_n1 --> n_w1 : HTTP/1.1 503 Service Unavailable
note over n_n2, n_n3 : Ko'pchilik tomon yangi lider saylaydi
n_n2 -> n_n3 : RequestVote n2 -> n3
n_n3 --> n_n2 : RequestVote reply
note over n_n2, n_n1 : Eski lider qaytadi va o'rnini bo'shatadi
n_n2 -> n_n1 : INFO n1: AppendEntries with term 2 > my term 1
note over n_w1, n_ledger : To'xtab qolgan worker eski lock bilan uyg'onadi
n_w1 -> n_ledger : UPDATE payout_runs SET fence = 42, status = 'running'
n_ledger --> n_w1 : UPDATE 0
@enduml

PlantUML: komponentlar

@startuml
left to right direction
collections "To'lov worker 1" as n_w1 #fff4d6;line:b26a00
collections "To'lov worker 2" as n_w2
database "To'lovlar bazasi" as n_ledger
component "Node 1" as n_n1
component "Node 2" as n_n2
component "Node 3" as n_n3
component "Node 4" as n_n4
component "Node 5" as n_n5
n_w1 -[#c62828]-> n_ledger
n_w2 --> n_ledger
n_n2 --> n_n3
n_n2 --> n_n4
n_w1 --> n_n2
n_n2 --> n_n1
n_n2 --> n_n5
@enduml

Xulosa

Lider bo'yicha kelishish uchun ping emas, ko'pchilik kerak.

1) 3 yoki 5 node ishlating: ular 1 yoki 2 nosozlikka chidaydi. 2) O'zgarish faqat ko'pchilik saqlaganda commit bo'ladi. 3) Term va ovozlar har term'da ko'pi bilan bitta lider beradi; ozchilik hech qachon commit qila olmaydi. 4) Lock fencing token olib yuradi, himoyalangan resurs uni tekshiradi. Buni o'zingiz yozmang: lock, lider va config'ni etcd, ZooKeeper yoki Consul'da saqlang.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Ikki marta to'langan48 000 $0 $
Lock yo'q vaqt0 s0,4 s
Lock yozish vaqti2 ms7 ms

Qadamlar

  1. Muammo Ikki worker, ikkalasi ham o'zini boshliq deb biladi

    Tungi to'lov job'i ishonchlilik uchun ikki serverda turadi.

  2. Yechim Uchinchi tomondan so'ra: bitta lock server

    Endi qarorni alohida lock server qabul qiladi.

  3. Muammo Lock server — yangi yagona nosozlik nuqtasi

    Lock serverning diski ishdan chiqdi.

  4. Yechim Besh node, bitta lider, bitta ko'chiriladigan log

    Endi lock servisi besh node'da ishlaydi.

  5. Holat Yozuv ko'pchilikda bo'lsa, xavfsiz

    Worker 1 lock so'raydi.

  6. Muammo Lider bitta follower bilan uzilib qoldi

    Switch ishdan chiqdi.

  7. Yechim Ko'pchilik tomon yangi lider saylaydi

    Node 2, 3 va 4 liderdan xabar olmay qoladi.

  8. Holat Eski lider qaytadi va o'rnini bo'shatadi

    Switch almashtirildi.

  9. Holat To'xtab qolgan worker eski lock bilan uyg'onadi

    Worker 1 uzoq garbage collection pauzasida 20 s qotib qoladi.

  10. Xulosa Xulosa

    Lider bo'yicha kelishish uchun ping emas, ko'pchilik kerak.

Atamalar

Konsensus (Raft)
Bir nechta node lider va umumiy log bo'yicha ko'pchilik ovozi bilan kelishadi. O'zgarish ko'pchilik saqlaganda commit bo'ladi, term va ovozlar har term'da ko'pi bilan bitta lider beradi.
Fencing token
Har yangi lock bilan beriladigan o'suvchi raqam. Himoyalangan resurs ko'rgan eng katta raqamni eslab qoladi va eski lock bilan uyg'ongan worker'ning yozuvini rad etadi.