mufassall

Konspekt · Arxitektura

Tranzaksion outbox

14 daq · Yuqori · mufassall.dtwin.uz/uz/l/outbox-pattern/

← Darsni ochish

Muammo

Do'kon xaridni o'z bazasiga saqlaydi, keyin loyallik servisi ball yozib qo'yishi uchun hodisa (event) e'lon qiladi. Ikki xil tizimga ikki alohida yozuv, umumiy tranzaksiya yo'q: ba'zida commit o'tadi-yu, publish o'tmaydi va ballar izsiz yo'qoladi. Outbox hodisani o'sha bazaga, o'sha tranzaksiyada yozadi, relay uni kamida bir marta yetkazadi, iste'molchi takrorlarni tashlaydi, qolganini esa kalit, tozalash va log'ni o'qish hal qiladi.

Nimani o'rganasiz

  • Nega "avval commit, keyin publish" qaysi tartibda bo'lmasin hodisani yo'qotadi va o'sha tranzaksiyadagi outbox jadvali bu bo'shliqni qanday yopadi
  • Relay outbox'ni FOR UPDATE SKIP LOCKED bilan qanday o'qiydi va nega u faqat at-least-once yetkazishni va'da qila oladi
  • Iste'molchi idempotentligi: ta'sir bilan birga commit bo'ladigan qayta ishlangan hodisalar jadvali
  • Xabar kaliti bilan har aggregate ichida tartib, yuborilgan qatorlarni tozalash va polling bilan log'ni o'qish o'rtasidagi tanlov
Yakuniy sahna
70%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_app(["Do'kon ilovasi"])
  n_api["Xarid servisi"]
  n_db[("Xaridlar bazasi")]
  n_broker[["Broker"]]
  n_loyalty["Loyallik servisi"]
  n_ldb[("Loyallik bazasi")]
  n_relay("Outbox relay")
  n_app -->|"chaqiruv"| n_api
  n_api --> n_db
  n_broker -->|"asinxron"| n_loyalty
  n_loyalty --> n_ldb
  n_relay --> n_db
  n_relay -->|"asinxron"| n_broker

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_api as Xarid servisi
  participant n_broker as Broker
  participant n_db as Xaridlar bazasi
  participant n_relay as Outbox relay
  participant n_loyalty as Loyallik servisi
  participant n_ldb as Loyallik bazasi
  Note over n_api,n_broker: Xaridni saqla, keyin hammaga xabar ber
  n_api->>n_broker: POST /v1/purchases -#62; purchase p_9051
  Note over n_api,n_broker: Commit o'tadi, publish o'tmaydi
  n_api->>n_broker: 21:14:07.120 p_9188 COMMIT ok
  Note over n_api,n_db: Hodisani o'sha tranzaksiyaning o'ziga yoz
  n_api->>n_db: BEGIN#59;
  n_db-->>n_api: COMMIT 3 ms (both rows or none)
  Note over n_relay,n_db: Relay outbox'dagini e'lon qiladi
  n_relay->>n_db: SELECT id, aggregate_id, type, payload FROM outbox
  Note over n_broker,n_loyalty: Relay qayta ishga tushadi va ball ikki marta yoziladi
  n_broker->>n_loyalty: 21:30:01 loyalty evt_01HV9Q PurchaseCompleted p_9188 +249 points
  Note over n_loyalty,n_ldb: Iste'molchi nimani qo'llaganini eslab qoladi
  n_loyalty->>n_ldb: BEGIN#59;
  n_ldb-->>n_loyalty: INSERT 0 0 -#62; duplicate, skipped
  Note over n_broker,n_loyalty: Qaytarish xariddan oldin yetib keladi
  n_broker->>n_loyalty: partition 3 offset 88120 PurchaseRefunded p_9188
  Note over n_relay,n_broker: Har xaridga bitta kalit, har kalitga bitta partition
  n_relay->>n_broker: produce topic=purchase-events
  n_broker-->>n_relay: ack partition 3 offset 88121
  Note over n_relay,n_db: Outbox har hodisani abadiy saqlaydi
  n_relay->>n_db: EXPLAIN ANALYZE SELECT ... FROM outbox WHERE sent_at IS NULL LIMIT 200#59;
  n_db-->>n_relay: Index Scan using outbox_unsent (rows=200)
  Note over n_relay,n_db: Yuborilgan qatorlarni kichik paketlarda o'chir
  n_relay->>n_db: DELETE FROM outbox
  n_db-->>n_relay: DELETE 5000
  Note over n_relay,n_db: Yoki outbox'ni bazaning log'idan o'qi
  n_relay->>n_db: relay mode=log slot outbox_slot

D2

direction: right

n_app: "Do'kon ilovasi" {
  shape: rectangle
}
n_api: "Xarid servisi" {
  shape: rectangle
}
n_db: "Xaridlar bazasi" {
  shape: cylinder
}
n_broker: "Broker" {
  shape: queue
}
n_loyalty: "Loyallik servisi" {
  shape: rectangle
}
n_ldb: "Loyallik bazasi" {
  shape: cylinder
}
n_relay: "Outbox relay" {
  shape: rectangle
  style.multiple: true
}

n_app -> n_api: "chaqiruv"
n_api -> n_db
n_broker -> n_loyalty: "asinxron"
n_loyalty -> n_ldb
n_relay -> n_db
n_relay -> n_broker: "asinxron"

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Xarid servisi" as n_api
queue "Broker" as n_broker
database "Xaridlar bazasi" as n_db
participant "Outbox relay" as n_relay
participant "Loyallik servisi" as n_loyalty
database "Loyallik bazasi" as n_ldb
note over n_api, n_broker : Xaridni saqla, keyin hammaga xabar ber
n_api ->> n_broker : POST /v1/purchases -> purchase p_9051
note over n_api, n_broker : Commit o'tadi, publish o'tmaydi
n_api ->> n_broker : 21:14:07.120 p_9188 COMMIT ok
note over n_api, n_db : Hodisani o'sha tranzaksiyaning o'ziga yoz
n_api -> n_db : BEGIN;
n_db --> n_api : COMMIT 3 ms (both rows or none)
note over n_relay, n_db : Relay outbox'dagini e'lon qiladi
n_relay -> n_db : SELECT id, aggregate_id, type, payload FROM outbox
note over n_broker, n_loyalty : Relay qayta ishga tushadi va ball ikki marta yoziladi
n_broker ->> n_loyalty : [dublikat] 21:30:01 loyalty evt_01HV9Q PurchaseCompleted p_9188 +249 points
note over n_loyalty, n_ldb : Iste'molchi nimani qo'llaganini eslab qoladi
n_loyalty -> n_ldb : BEGIN;
n_ldb --> n_loyalty : INSERT 0 0 -> duplicate, skipped
note over n_broker, n_loyalty : Qaytarish xariddan oldin yetib keladi
n_broker ->> n_loyalty : partition 3 offset 88120 PurchaseRefunded p_9188
note over n_relay, n_broker : Har xaridga bitta kalit, har kalitga bitta partition
n_relay ->> n_broker : produce topic=purchase-events
n_broker -->> n_relay : ack partition 3 offset 88121
note over n_relay, n_db : Outbox har hodisani abadiy saqlaydi
n_relay -> n_db : EXPLAIN ANALYZE SELECT ... FROM outbox WHERE sent_at IS NULL LIMIT 200;
n_db --> n_relay : Index Scan using outbox_unsent (rows=200)
note over n_relay, n_db : Yuborilgan qatorlarni kichik paketlarda o'chir
n_relay -> n_db : DELETE FROM outbox
n_db --> n_relay : DELETE 5000
note over n_relay, n_db : Yoki outbox'ni bazaning log'idan o'qi
n_relay -> n_db : relay mode=log slot outbox_slot
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Do'kon ilovasi" as n_app
component "Xarid servisi" as n_api
database "Xaridlar bazasi" as n_db
queue "Broker" as n_broker
component "Loyallik servisi" as n_loyalty
database "Loyallik bazasi" as n_ldb
collections "Outbox relay" as n_relay
n_app --> n_api : chaqiruv
n_api --> n_db
n_broker --> n_loyalty : asinxron
n_loyalty --> n_ldb
n_relay --> n_db
n_relay --> n_broker : asinxron
@enduml

Xulosa

Outbox uchun olti qoida.

1) Baza va broker'ga hech qachon ikki qadamda yozmang: qaysi tartibda bo'lmasin, hodisa yo yo'qoladi, yo to'qib chiqariladi. 2) Hodisani ma'lumot bilan bitta tranzaksiyada outbox jadvaliga qo'shing. 3) Relay yuborilmagan qatorlarni e'lon qiladi va ack'dan keyin belgilaydi; bu at-least-once, hech qachon exactly-once emas. 4) Iste'molchilar event id bo'yicha, ta'sir bilan bitta tranzaksiyada dedup qiladi. 5) Xabarlarga aggregate id'ni kalit qiling va relay'larni kalit bo'yicha bo'ling, shunda bitta aggregate hodisalari tartibda qoladi. 6) Yuborilgan qatorlarni o'chiring yoki partition'ini tashlang; soddalik uchun polling'ni, past kechikish uchun log'ni o'qishni tanlang.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Yo'qolgan hodisalar0/kun0/kun
Ikki marta yozilgan ball0/kun0/kun
Tartibi buzilgan hodisalar0/kun0/kun
Commit'dan publish'gacha4 ms40 ms
Yuborilmagan outbox qatorlari00
Outbox'dagi qatorlar0900 000

Qadamlar

  1. Holat Xaridni saqla, keyin hammaga xabar ber

    Xarid servisi xaridni o'z bazasiga commit qiladi.

  2. Muammo Commit o'tadi, publish o'tmaydi

    21:14 da broker sekinlashadi.

  3. Yechim Hodisani o'sha tranzaksiyaning o'ziga yoz

    Servis broker bilan gaplashishni to'xtatadi.

  4. Yechim Relay outbox'dagini e'lon qiladi

    Kichik worker, relay, har 500 ms da yuborilmagan qatorlarni o'qiydi, ularni e'lon qiladi, broker'dan ack kutadi va qatorlarni yuborilgan deb belgilaydi.

  5. Muammo Relay qayta ishga tushadi va ball ikki marta yoziladi

    Relay pod'i publish bilan belgilash orasida o'ldiriladi.

  6. Yechim Iste'molchi nimani qo'llaganini eslab qoladi

    Loyallik servisi har event_id ni ball yozish bilan bitta tranzaksiyada processed_events ga yozadi.

  7. Muammo Qaytarish xariddan oldin yetib keladi

    O'tkazuvchanlik uchun ikkinchi relay pod'i qo'shiladi.

  8. Yechim Har xaridga bitta kalit, har kalitga bitta partition

    Relay xabar kalitini xarid id'siga tenglaydi, har relay pod'i esa faqat hash'i o'z ulushiga tushgan kalitlarni oladi.

  9. Muammo Outbox har hodisani abadiy saqlaydi

    Yuborilgan qatorlarni hech kim o'chirmaydi.

  10. Yechim Yuborilgan qatorlarni kichik paketlarda o'chir

    Alohida ish har daqiqada 3 kundan oldin yuborilgan qatorlarni 5 000 tadan o'chiradi.

  11. Yechim Yoki outbox'ni bazaning log'idan o'qi

    Polling o'rniga relay bazaning write-ahead log'ini replikatsiya slot'i orqali kuzatib borishi mumkin, xuddi CDC kabi.

  12. Xulosa Xulosa

    Outbox uchun olti qoida.

Atamalar

Outbox relay
Outbox jadvalidagi yuborilmagan qatorlarni o'qib, broker'ga e'lon qiladigan va ack'dan keyin ularni yuborilgan deb belgilaydigan worker. Belgilashdan oldin yiqilsa, qatorlarni yana yuboradi, shuning uchun yetkazish at-least-once bo'ladi.
Inbox (processed events) jadvali
Iste'molchi qo'llagan har hodisa id'sini unique kalit bilan, ta'sir bilan bitta tranzaksiyada yozadigan jadval. Takroriy yetkazish kalitga uriladi va o'tkazib yuboriladi.
Xabar kaliti (partition key)
Broker xabarni qaysi partition'ga qo'yishini belgilaydigan qiymat, odatda aggregate id. Bir xil kalitli xabarlar bitta partition'ga tushadi va tartibi saqlanadi; turli kalitlar orasida tartib kafolatlanmaydi.
Aggregate versiyasi
Hodisaning o'z aggregate'i ichidagi tartib raqami: 1, 2, 3. Iste'molchi oxirgi qo'llangan versiyani saqlaydi, eskirog'ini tashlaydi, oraliq bo'lsa oldingisini kutadi.