mufassall

Konspekt · Arxitektura

O'zgarishlarni ushlash (CDC)

15 daq · Yuqori · mufassall.dtwin.uz/uz/l/cdc/

← Darsni ochish

Muammo

Internet-do'kon buyurtmalarni SQL bazada saqlaydi va har o'zgarishni qidiruv indeksi va ma'lumotlar omboriga ko'chirishi kerak. Bitta koddan bazaga ham, hodisalar jurnaliga ham yozish hodisalarni yo'qotadi. Bazaning o'z o'zgarishlar jurnalini o'qish esa yo'qotmaydi, agar tartib, qayta ishga tushish, birinchi snapshot va sxema o'zgarishlari o'ylab qo'yilsa.

Nimani o'rganasiz

  • Nega bazaga va hodisalar jurnaliga qo'sh yozuv ajralib ketadi
  • CDC konnektor WAL yoki binlog'ni qanday o'qiydi va o'zgarish hodisasi qanday ko'rinadi
  • Birlamchi kalit bo'yicha tartib, saqlangan offset va qayta ishga tushgandan keyingi takrorlar
  • Boshlang'ich snapshot va oqim, replikatsiya sloti egallagan disk, sxema o'zgarishi va outbox
Yakuniy sahna
40%60%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_client(["Do'kon ilovasi"])
  n_api["Buyurtmalar API"]
  n_db[("Buyurtmalar bazasi")]
  n_topic[["Hodisalar jurnali"]]
  n_search["Qidiruv indeksi"]
  n_connector("CDC konnektor")
  n_warehouse[("Ma'lumotlar ombori")]
  n_client --> n_api
  n_api --> n_db
  n_topic --> n_search
  n_db --> n_connector
  n_connector --> n_topic
  n_topic --> n_warehouse

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_api as Buyurtmalar API
  participant n_topic as Hodisalar jurnali
  participant n_db as Buyurtmalar bazasi
  participant n_connector as CDC konnektor
  participant n_search as Qidiruv indeksi
  participant n_warehouse as Ma'lumotlar ombori
  Note over n_api,n_topic: API ikki marta yozadi: bazaga va hodisalar jurnaliga
  n_api->>n_topic: PRODUCE orders-events
  n_topic-->>n_api: OK partition 3, offset 88123
  Note over n_api,n_topic: Commit o'tdi, e'lon o'tmadi
  n_api-xn_topic: 10:00:01.120 COMMIT order 5521 status=paid
  Note over n_db,n_connector: Bazaning o'z o'zgarishlar jurnalini o'qiymiz
  n_db->>n_connector: START_REPLICATION SLOT shop_cdc LOGICAL 0/1A2B3C0
  Note over n_connector,n_topic: O'zgarish hodisasi nimani olib yuradi
  n_connector->>n_topic: PRODUCE shop.public.orders
  Note over n_topic,n_search: Bitta buyurtmaning ikki o'zgarishi noto'g'ri tartibda keladi
  n_topic->>n_search: search consumer, 2 threads, round-robin partitions
  Note over n_connector,n_topic: Birlamchi kalit bo'yicha partitsiya, eski versiyalarni tashlash
  n_connector->>n_topic: connector config: orders
  Note over n_connector,n_topic: Konnektor qayta ishga tushadi va o'zgarishlarni yana yuboradi
  n_connector->>n_topic: connector restarted, stored offset lsn=0/1A2B000
  Note over n_db,n_connector: Konnektor ishlamayapti, baza diski to'lib boryapti
  n_db-xn_connector: pg_replication_slots: shop_cdc active=false since Saturday
  Note over n_db,n_connector: Yangi iste'molchi: avval snapshot, keyin oqim
  n_db->>n_connector: BEGIN ISOLATION LEVEL REPEATABLE READ#59;
  Note over n_topic,n_warehouse: Ustun nomini o'zgartirish iste'molchini sindiradi
  n_topic->>n_warehouse: ALTER TABLE orders RENAME COLUMN total TO total_amount#59;
  n_warehouse-->>n_topic: warehouse: field total missing in 18400 events
  Note over n_api,n_db: Outbox va CDC: jadvallarni emas, hodisalarni e'lon qilamiz
  n_api->>n_db: BEGIN#59;

D2

direction: right

n_client: "Do'kon ilovasi" {
  shape: rectangle
}
n_api: "Buyurtmalar API" {
  shape: rectangle
}
n_db: "Buyurtmalar bazasi" {
  shape: cylinder
}
n_topic: "Hodisalar jurnali" {
  shape: queue
}
n_search: "Qidiruv indeksi" {
  shape: rectangle
}
n_connector: "CDC konnektor" {
  shape: rectangle
  style.multiple: true
}
n_warehouse: "Ma'lumotlar ombori" {
  shape: cylinder
}

n_client -> n_api
n_api -> n_db
n_topic -> n_search
n_db -> n_connector
n_connector -> n_topic
n_topic -> n_warehouse

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Buyurtmalar API" as n_api
queue "Hodisalar jurnali" as n_topic
database "Buyurtmalar bazasi" as n_db
participant "CDC konnektor" as n_connector
participant "Qidiruv indeksi" as n_search
database "Ma'lumotlar ombori" as n_warehouse
note over n_api, n_topic : API ikki marta yozadi: bazaga va hodisalar jurnaliga
n_api -> n_topic : PRODUCE orders-events
n_topic --> n_api : OK partition 3, offset 88123
note over n_api, n_topic : Commit o'tdi, e'lon o'tmadi
n_api ->x n_topic : [yo'qoldi] 10:00:01.120 COMMIT order 5521 status=paid
note over n_db, n_connector : Bazaning o'z o'zgarishlar jurnalini o'qiymiz
n_db -> n_connector : START_REPLICATION SLOT shop_cdc LOGICAL 0/1A2B3C0
note over n_connector, n_topic : O'zgarish hodisasi nimani olib yuradi
n_connector -> n_topic : PRODUCE shop.public.orders
note over n_topic, n_search : Bitta buyurtmaning ikki o'zgarishi noto'g'ri tartibda keladi
n_topic -> n_search : [dublikat] search consumer, 2 threads, round-robin partitions
note over n_connector, n_topic : Birlamchi kalit bo'yicha partitsiya, eski versiyalarni tashlash
n_connector -> n_topic : connector config: orders
note over n_connector, n_topic : Konnektor qayta ishga tushadi va o'zgarishlarni yana yuboradi
n_connector -> n_topic : [dublikat] connector restarted, stored offset lsn=0/1A2B000
note over n_db, n_connector : Konnektor ishlamayapti, baza diski to'lib boryapti
n_db ->x n_connector : [yo'qoldi] pg_replication_slots: shop_cdc active=false since Saturday
note over n_db, n_connector : Yangi iste'molchi: avval snapshot, keyin oqim
n_db -> n_connector : BEGIN ISOLATION LEVEL REPEATABLE READ;
note over n_topic, n_warehouse : Ustun nomini o'zgartirish iste'molchini sindiradi
n_topic -> n_warehouse : ALTER TABLE orders RENAME COLUMN total TO total_amount;
n_warehouse --> n_topic : warehouse: field total missing in 18400 events
note over n_api, n_db : Outbox va CDC: jadvallarni emas, hodisalarni e'lon qilamiz
n_api -> n_db : BEGIN;
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Do'kon ilovasi" as n_client
component "Buyurtmalar API" as n_api
database "Buyurtmalar bazasi" as n_db
queue "Hodisalar jurnali" as n_topic
component "Qidiruv indeksi" as n_search
collections "CDC konnektor" as n_connector
database "Ma'lumotlar ombori" as n_warehouse
n_client --> n_api
n_api --> n_db
n_topic --> n_search
n_db --> n_connector
n_connector --> n_topic
n_topic --> n_warehouse
@enduml

Xulosa

O'zgarishlarni ushlash uchun oltita qoida.

1) Bazaga va hodisalar jurnaliga hech qachon ikki alohida qadam bilan yozmang. 2) Konnektor bazaning o'z jurnalini o'qisin, shunda commit qilingan o'zgarish doim e'lon qilinadi. 3) Hodisa kaliti birlamchi kalit: tartib qator uchun saqlanadi, butun topic uchun emas. 4) Yetkazish kamida bir marta; iste'molchilar takrorni jurnaldagi o'rin bo'yicha tashlaydi. 5) Replikatsiya slotini kuzating, aks holda baza diski to'ladi. 6) Izchil snapshot bilan boshlang, keyin oqim; sxemani shartnoma deb biling yoki outbox orqali e'lon qiling.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Qidiruvda yo'q buyurtma00
Qidiruv orqada qolishi1 s2 s
Ushlab turilgan WAL2 GB3 GB
Takror hodisalar00

Qadamlar

  1. Holat API ikki marta yozadi: bazaga va hodisalar jurnaliga

    Buyurtma to'langanda Buyurtmalar API bazadagi qatorni yangilaydi, keyin qidiruv indeksi yangi holatni ko'rsatishi uchun hodisa e'lon qiladi.

  2. Muammo Commit o'tdi, e'lon o'tmadi

    Baza paid ni saqlaydi.

  3. Yechim Bazaning o'z o'zgarishlar jurnalini o'qiymiz

    Baza har commit qilingan o'zgarishni hamma narsadan oldin o'z jurnaliga yozadi: PostgreSQL'da WAL, MySQL'da binlog.

  4. Holat O'zgarish hodisasi nimani olib yuradi

    Har hodisa qaysi qator o'zgarganini, qanday (c yaratish, u yangilash, d o'chirish), qatorning oldingi va keyingi ko'rinishini va o'zgarish jurnalning qayerida turganini aytadi.

  5. Muammo Bitta buyurtmaning ikki o'zgarishi noto'g'ri tartibda keladi

    Konnektor hodisalarni partitsiyalarga navbatma-navbat tarqatadi, qidiruv iste'molchisi esa partitsiyalarni parallel o'qiydi.

  6. Yechim Birlamchi kalit bo'yicha partitsiya, eski versiyalarni tashlash

    Konnektor endi birlamchi kalitni xabar kaliti qilib oladi.

  7. Holat Konnektor qayta ishga tushadi va o'zgarishlarni yana yuboradi

    Konnektor o'z o'rnini, offset'ni, har 10 soniyada saqlaydi.

  8. Muammo Konnektor ishlamayapti, baza diski to'lib boryapti

    Konnektor shanba kuni to'xtadi va hech kim sezmadi.

  9. Yechim Yangi iste'molchi: avval snapshot, keyin oqim

    Konnektor yana ishlayapti va ma'lumotlar ombori qo'shiladi.

  10. Muammo Ustun nomini o'zgartirish iste'molchini sindiradi

    Dasturchi total ni total_amount ga o'zgartiradi.

  11. Yechim Outbox va CDC: jadvallarni emas, hodisalarni e'lon qilamiz

    Xom jadvallarni oqim qilish o'rniga API o'zgarish bilan bitta tranzaksiyada outbox jadvaliga biznes hodisasini yozadi.

  12. Xulosa Xulosa

    O'zgarishlarni ushlash uchun oltita qoida.

Atamalar

CDC (o'zgarishlarni ushlash)
Bazaning o'z jurnalini (WAL yoki binlog) o'qib, har bir yozuv o'zgarishini hodisa sifatida tashqariga uzatish. Ilova ikki joyga yozmaydi, shuning uchun baza va hodisalar bir-biridan ajralib ketmaydi.
Ikki joyga yozish (dual write)
Ilova bir o'zgarishni ketma-ket bazaga va brokerga (yoki qidiruv indeksiga) yozadi. Ikkinchi yozuv yiqilsa yoki tartib aralashsa, ikki tizim jimgina ajralib ketadi.