mufassall

Konspekt · Real tajribalar

Tajriba: Discord xabarlarni qanday saqlaydi

16 daq · Yuqori · mufassall.dtwin.uz/uz/l/case-discord-messages/

← Darsni ochish

Muammo

Bitta MongoDB kolleksiyasidan Cassandra'ga, keyin ScyllaDB'ga: Discord trillionlab chat xabarlarini qanday o'qiladigan holda saqladi. Partition kaliti, tombstone, issiq partitionlar, so'rovlarni birlashtiradigan data servislar va soniyasiga 3,2 million xabar tezlikdagi migratsiya. Manbalar: https://discord.com/blog/how-discord-stores-billions-of-messages va https://discord.com/blog/how-discord-stores-trillions-of-messages

Nimani o'rganasiz

  • Nega (kanal, vaqt bucket'i) partition kaliti partitionlarni kichik saqlaydi
  • O'chirish qanday tombstone'ga aylanadi va o'qishda u nimaga tushadi
  • Issiq partition: bitta band kanal butun node'ni sekinlatadi
  • Data servislar: so'rovlarni birlashtirish va kanalning consistent hash'i bo'yicha yo'naltirish
  • Katta jonli migratsiya qanday tez qilindi, tekshirildi va oxiriga yetkazildi
Yakuniy sahna
30%40%40%40%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_users(["Foydalanuvchilar"])
  n_api["API monolit"]
  n_ds["Data servis A"]
  n_ds2["Data servis B"]
  n_scylla[("ScyllaDB")]
  n_users --> n_api
  n_api --> n_ds
  n_api --> n_ds2
  n_ds --> n_scylla
  n_ds2 --> n_scylla

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_api as API monolit
  participant n_mongo as MongoDB
  participant n_n2 as Cassandra node 2
  participant n_n1 as Cassandra node 1
  participant n_ds as Data servis A
  participant n_ds2 as Data servis B
  participant n_mig as Rust migrator
  participant n_scylla as ScyllaDB
  Note over n_api,n_mongo: 2015: bitta kolleksiya, bitta indeks
  n_api->>n_mongo: db.messages.find({ channel_id: 1234 }).sort({ created_at: -1 }).limit(50)
  Note over n_api,n_n2: Cassandra: kanal vaqt bucket'lariga bo'lingan
  n_api->>n_n2: CREATE TABLE messages (
  Note over n_api,n_n2: O'chirilgan xabarlar ham o'qishga tushadi
  n_api->>n_n2: SELECT * FROM messages WHERE channel_id = 1234 AND bucket = 41 LIMIT 50#59;
  n_n2-->>n_api: 1 row (about 20 s)
  Note over n_api,n_n1: 12 node'dan 177 tagacha
  n_api->>n_n1: cluster cassandra-messages, early 2022
  Note over n_api,n_n2: Bitta band kanal butun node'ni sekinlatadi
  n_api->>n_n2: CONSISTENCY QUORUM#59;
  n_n2-->>n_api: slow: one (channel_id, bucket) partition, one set of replicas
  Note over n_api,n_n1: Compaction ortda qoladi, GC pauzalari o'sadi
  n_api->>n_n1: n1: pending compactions keep growing
  Note over n_api,n_ds: API va baza orasida data servis
  n_api->>n_ds: gRPC: get messages (simplified)
  Note over n_ds,n_n2: So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov
  n_ds->>n_n2: many users ask for the same row at the same time
  Note over n_api,n_ds2: Kanal bo'yicha consistent hash bilan yo'naltirish
  n_api->>n_ds2: routing key: channel_id
  Note over n_mig,n_scylla: Trillionlab xabarni ScyllaDB'ga ko'chirish
  n_mig->>n_scylla: migrator (Rust): token range done, checkpoint saved to SQLite
  Note over n_ds,n_scylla: Kamroq node, tekis kechikish
  n_ds->>n_scylla: after the switch to ScyllaDB (May 2022)

D2

direction: right

n_users: "Foydalanuvchilar" {
  shape: rectangle
}
n_api: "API monolit" {
  shape: rectangle
}
n_ds: "Data servis A" {
  shape: rectangle
}
n_ds2: "Data servis B" {
  shape: rectangle
}
n_scylla: "ScyllaDB" {
  shape: cylinder
}

n_users -> n_api
n_api -> n_ds
n_api -> n_ds2
n_ds -> n_scylla
n_ds2 -> n_scylla

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "API monolit" as n_api
database "MongoDB" as n_mongo
database "Cassandra node 2" as n_n2
database "Cassandra node 1" as n_n1
participant "Data servis A" as n_ds
participant "Data servis B" as n_ds2
participant "Rust migrator" as n_mig
database "ScyllaDB" as n_scylla
note over n_api, n_mongo : 2015: bitta kolleksiya, bitta indeks
n_api -> n_mongo : db.messages.find({ channel_id: 1234 }).sort({ created_at: -1 }).limit(50)
note over n_api, n_n2 : Cassandra: kanal vaqt bucket'lariga bo'lingan
n_api -> n_n2 : CREATE TABLE messages (
note over n_api, n_n2 : O'chirilgan xabarlar ham o'qishga tushadi
n_api -> n_n2 : SELECT * FROM messages WHERE channel_id = 1234 AND bucket = 41 LIMIT 50;
n_n2 --> n_api : 1 row (about 20 s)
note over n_api, n_n1 : 12 node'dan 177 tagacha
n_api -> n_n1 : cluster cassandra-messages, early 2022
note over n_api, n_n2 : Bitta band kanal butun node'ni sekinlatadi
n_api -> n_n2 : CONSISTENCY QUORUM;
n_n2 --> n_api : slow: one (channel_id, bucket) partition, one set of replicas
note over n_api, n_n1 : Compaction ortda qoladi, GC pauzalari o'sadi
n_api -> n_n1 : n1: pending compactions keep growing
note over n_api, n_ds : API va baza orasida data servis
n_api -> n_ds : gRPC: get messages (simplified)
note over n_ds, n_n2 : So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov
n_ds -> n_n2 : many users ask for the same row at the same time
note over n_api, n_ds2 : Kanal bo'yicha consistent hash bilan yo'naltirish
n_api -> n_ds2 : routing key: channel_id
note over n_mig, n_scylla : Trillionlab xabarni ScyllaDB'ga ko'chirish
n_mig -> n_scylla : migrator (Rust): token range done, checkpoint saved to SQLite
note over n_ds, n_scylla : Kamroq node, tekis kechikish
n_ds -> n_scylla : after the switch to ScyllaDB (May 2022)
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Foydalanuvchilar" as n_users
component "API monolit" as n_api
component "Data servis A" as n_ds
component "Data servis B" as n_ds2
database "ScyllaDB" as n_scylla
n_users --> n_api
n_api --> n_ds
n_api --> n_ds2
n_ds --> n_scylla
n_ds2 --> n_scylla
@enduml

Xulosa

Discord tarixidan nimani olish kerak.

1) Partition kalitini qanday o'qishingizga qarab tanlang va uni chegaralang: kanal va vaqt bucket'i. 2) O'chirish ham yozuv; tombstone va compaction o'qish narxiga kiradi. 3) Notekis kalitlar issiq partition beradi; bazani oldidagi qatlam bilan himoya qiling. 4) Bir xil o'qishlarni birlashtiring va ular uchrashishi uchun o'sha kalit bo'yicha yo'naltiring. 5) Katta migratsiyada nusxalovchini tez va davom ettiriladigan qiling, almashtirishdan oldin ikkala ombordan o'qishni solishtiring.

Manbalar: https://discord.com/blog/how-discord-stores-billions-of-messages va https://discord.com/blog/how-discord-stores-trillions-of-messages

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
O'qish p995 ms15 ms
Yozish p9970 ms5 ms
Baza node'lari1272

Qadamlar

  1. Holat 2015: bitta kolleksiya, bitta indeks

    Discord avval hamma xabarni bitta MongoDB kolleksiyasida, channel_id va created_at bo'yicha qo'shma indeks bilan saqlagan.

  2. Yechim Cassandra: kanal vaqt bucket'lariga bo'lingan

    Discord xabarlarni Cassandra'ga ko'chirdi.

  3. Muammo O'chirilgan xabarlar ham o'qishga tushadi

    Cassandra'da o'chirish ham yozuv: eski qiymatni compaction ikkalasini tozalaguncha yashirib turadigan tombstone.

  4. Holat 12 node'dan 177 tagacha

    2022-yil boshiga kelib xabarlar klasterida 177 node va trillionlab xabar bor edi.

  5. Muammo Bitta band kanal butun node'ni sekinlatadi

    Katta server e'lon joylaydi va juda ko'p odam bir vaqtda bitta kanalni ochadi.

  6. Muammo Compaction ortda qoladi, GC pauzalari o'sadi

    Cassandra fon rejimida o'z fayllarini birlashtiradi (compaction).

  7. Yechim API va baza orasida data servis

    Discord API monolit va baza klasterlari orasiga data servislar qo'ydi.

  8. Yechim So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov

    Ko'p foydalanuvchi bir vaqtda bitta qatorni so'rasa, data servis bazaga faqat bir marta boradi.

  9. Yechim Kanal bo'yicha consistent hash bilan yo'naltirish

    So'rovlar kalit bo'yicha yo'naltiriladi; xabarlar uchun kalit — kanal id'si.

  10. Yechim Trillionlab xabarni ScyllaDB'ga ko'chirish

    ScyllaDB Cassandra bilan mos, C++'da yozilgan, axlat yig'uvchisi yo'q va ishni CPU yadrolari bo'yicha bo'ladi.

  11. Holat Kamroq node, tekis kechikish

    Discord 2022-yil mayida ScyllaDB'ga o'tdi.

  12. Xulosa Xulosa

    Discord tarixidan nimani olish kerak.