Konspekt · Real tajribalar
Tajriba: Discord xabarlarni qanday saqlaydi
Muammo
Bitta MongoDB kolleksiyasidan Cassandra'ga, keyin ScyllaDB'ga: Discord trillionlab chat xabarlarini qanday o'qiladigan holda saqladi. Partition kaliti, tombstone, issiq partitionlar, so'rovlarni birlashtiradigan data servislar va soniyasiga 3,2 million xabar tezlikdagi migratsiya. Manbalar: https://discord.com/blog/how-discord-stores-billions-of-messages va https://discord.com/blog/how-discord-stores-trillions-of-messages
Nimani o'rganasiz
- Nega (kanal, vaqt bucket'i) partition kaliti partitionlarni kichik saqlaydi
- O'chirish qanday tombstone'ga aylanadi va o'qishda u nimaga tushadi
- Issiq partition: bitta band kanal butun node'ni sekinlatadi
- Data servislar: so'rovlarni birlashtirish va kanalning consistent hash'i bo'yicha yo'naltirish
- Katta jonli migratsiya qanday tez qilindi, tekshirildi va oxiriga yetkazildi
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_users(["Foydalanuvchilar"])
n_api["API monolit"]
n_ds["Data servis A"]
n_ds2["Data servis B"]
n_scylla[("ScyllaDB")]
n_users --> n_api
n_api --> n_ds
n_api --> n_ds2
n_ds --> n_scylla
n_ds2 --> n_scyllaMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_api as API monolit
participant n_mongo as MongoDB
participant n_n2 as Cassandra node 2
participant n_n1 as Cassandra node 1
participant n_ds as Data servis A
participant n_ds2 as Data servis B
participant n_mig as Rust migrator
participant n_scylla as ScyllaDB
Note over n_api,n_mongo: 2015: bitta kolleksiya, bitta indeks
n_api->>n_mongo: db.messages.find({ channel_id: 1234 }).sort({ created_at: -1 }).limit(50)
Note over n_api,n_n2: Cassandra: kanal vaqt bucket'lariga bo'lingan
n_api->>n_n2: CREATE TABLE messages (
Note over n_api,n_n2: O'chirilgan xabarlar ham o'qishga tushadi
n_api->>n_n2: SELECT * FROM messages WHERE channel_id = 1234 AND bucket = 41 LIMIT 50#59;
n_n2-->>n_api: 1 row (about 20 s)
Note over n_api,n_n1: 12 node'dan 177 tagacha
n_api->>n_n1: cluster cassandra-messages, early 2022
Note over n_api,n_n2: Bitta band kanal butun node'ni sekinlatadi
n_api->>n_n2: CONSISTENCY QUORUM#59;
n_n2-->>n_api: slow: one (channel_id, bucket) partition, one set of replicas
Note over n_api,n_n1: Compaction ortda qoladi, GC pauzalari o'sadi
n_api->>n_n1: n1: pending compactions keep growing
Note over n_api,n_ds: API va baza orasida data servis
n_api->>n_ds: gRPC: get messages (simplified)
Note over n_ds,n_n2: So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov
n_ds->>n_n2: many users ask for the same row at the same time
Note over n_api,n_ds2: Kanal bo'yicha consistent hash bilan yo'naltirish
n_api->>n_ds2: routing key: channel_id
Note over n_mig,n_scylla: Trillionlab xabarni ScyllaDB'ga ko'chirish
n_mig->>n_scylla: migrator (Rust): token range done, checkpoint saved to SQLite
Note over n_ds,n_scylla: Kamroq node, tekis kechikish
n_ds->>n_scylla: after the switch to ScyllaDB (May 2022)D2
direction: right
n_users: "Foydalanuvchilar" {
shape: rectangle
}
n_api: "API monolit" {
shape: rectangle
}
n_ds: "Data servis A" {
shape: rectangle
}
n_ds2: "Data servis B" {
shape: rectangle
}
n_scylla: "ScyllaDB" {
shape: cylinder
}
n_users -> n_api
n_api -> n_ds
n_api -> n_ds2
n_ds -> n_scylla
n_ds2 -> n_scyllaPlantUML: so'rovlar ketma-ketligi
@startuml
participant "API monolit" as n_api
database "MongoDB" as n_mongo
database "Cassandra node 2" as n_n2
database "Cassandra node 1" as n_n1
participant "Data servis A" as n_ds
participant "Data servis B" as n_ds2
participant "Rust migrator" as n_mig
database "ScyllaDB" as n_scylla
note over n_api, n_mongo : 2015: bitta kolleksiya, bitta indeks
n_api -> n_mongo : db.messages.find({ channel_id: 1234 }).sort({ created_at: -1 }).limit(50)
note over n_api, n_n2 : Cassandra: kanal vaqt bucket'lariga bo'lingan
n_api -> n_n2 : CREATE TABLE messages (
note over n_api, n_n2 : O'chirilgan xabarlar ham o'qishga tushadi
n_api -> n_n2 : SELECT * FROM messages WHERE channel_id = 1234 AND bucket = 41 LIMIT 50;
n_n2 --> n_api : 1 row (about 20 s)
note over n_api, n_n1 : 12 node'dan 177 tagacha
n_api -> n_n1 : cluster cassandra-messages, early 2022
note over n_api, n_n2 : Bitta band kanal butun node'ni sekinlatadi
n_api -> n_n2 : CONSISTENCY QUORUM;
n_n2 --> n_api : slow: one (channel_id, bucket) partition, one set of replicas
note over n_api, n_n1 : Compaction ortda qoladi, GC pauzalari o'sadi
n_api -> n_n1 : n1: pending compactions keep growing
note over n_api, n_ds : API va baza orasida data servis
n_api -> n_ds : gRPC: get messages (simplified)
note over n_ds, n_n2 : So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov
n_ds -> n_n2 : many users ask for the same row at the same time
note over n_api, n_ds2 : Kanal bo'yicha consistent hash bilan yo'naltirish
n_api -> n_ds2 : routing key: channel_id
note over n_mig, n_scylla : Trillionlab xabarni ScyllaDB'ga ko'chirish
n_mig -> n_scylla : migrator (Rust): token range done, checkpoint saved to SQLite
note over n_ds, n_scylla : Kamroq node, tekis kechikish
n_ds -> n_scylla : after the switch to ScyllaDB (May 2022)
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Foydalanuvchilar" as n_users
component "API monolit" as n_api
component "Data servis A" as n_ds
component "Data servis B" as n_ds2
database "ScyllaDB" as n_scylla
n_users --> n_api
n_api --> n_ds
n_api --> n_ds2
n_ds --> n_scylla
n_ds2 --> n_scylla
@endumlXulosa
Discord tarixidan nimani olish kerak.
1) Partition kalitini qanday o'qishingizga qarab tanlang va uni chegaralang: kanal va vaqt bucket'i. 2) O'chirish ham yozuv; tombstone va compaction o'qish narxiga kiradi. 3) Notekis kalitlar issiq partition beradi; bazani oldidagi qatlam bilan himoya qiling. 4) Bir xil o'qishlarni birlashtiring va ular uchrashishi uchun o'sha kalit bo'yicha yo'naltiring. 5) Katta migratsiyada nusxalovchini tez va davom ettiriladigan qiling, almashtirishdan oldin ikkala ombordan o'qishni solishtiring.
Manbalar: https://discord.com/blog/how-discord-stores-billions-of-messages va https://discord.com/blog/how-discord-stores-trillions-of-messages
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| O'qish p99 | 5 ms | 15 ms |
| Yozish p99 | 70 ms | 5 ms |
| Baza node'lari | 12 | 72 |
Qadamlar
- Holat 2015: bitta kolleksiya, bitta indeks
Discord avval hamma xabarni bitta MongoDB kolleksiyasida,
channel_idvacreated_atbo'yicha qo'shma indeks bilan saqlagan. - Yechim Cassandra: kanal vaqt bucket'lariga bo'lingan
Discord xabarlarni Cassandra'ga ko'chirdi.
- Muammo O'chirilgan xabarlar ham o'qishga tushadi
Cassandra'da o'chirish ham yozuv: eski qiymatni compaction ikkalasini tozalaguncha yashirib turadigan tombstone.
- Holat 12 node'dan 177 tagacha
2022-yil boshiga kelib xabarlar klasterida 177 node va trillionlab xabar bor edi.
- Muammo Bitta band kanal butun node'ni sekinlatadi
Katta server e'lon joylaydi va juda ko'p odam bir vaqtda bitta kanalni ochadi.
- Muammo Compaction ortda qoladi, GC pauzalari o'sadi
Cassandra fon rejimida o'z fayllarini birlashtiradi (compaction).
- Yechim API va baza orasida data servis
Discord API monolit va baza klasterlari orasiga data servislar qo'ydi.
- Yechim So'rovlarni birlashtirish: ko'p o'quvchiga bitta so'rov
Ko'p foydalanuvchi bir vaqtda bitta qatorni so'rasa, data servis bazaga faqat bir marta boradi.
- Yechim Kanal bo'yicha consistent hash bilan yo'naltirish
So'rovlar kalit bo'yicha yo'naltiriladi; xabarlar uchun kalit — kanal id'si.
- Yechim Trillionlab xabarni ScyllaDB'ga ko'chirish
ScyllaDB Cassandra bilan mos, C++'da yozilgan, axlat yig'uvchisi yo'q va ishni CPU yadrolari bo'yicha bo'ladi.
- Holat Kamroq node, tekis kechikish
Discord 2022-yil mayida ScyllaDB'ga o'tdi.
- Xulosa Xulosa
Discord tarixidan nimani olish kerak.