Case studies / 1 / 2
Case: how Discord stores messages
From one MongoDB collection to Cassandra and then to ScyllaDB: how Discord kept trillions of chat messages readable. Partition keys, tombstones, hot partitions, data services with request coalescing, and a migration at 3.2 million messages a second. Sources: https://discord.com/blog/how-discord-stores-billions-of-messages and https://discord.com/blog/how-discord-stores-trillions-of-messages
- 16 min
- Advanced
- 12 steps
- Read p99
- —
- Insert p99
- —
- DB nodes
- —
- request
- reply
- redelivery
- duplicate
- lost
Numbers are a model scenario, not measurements
The real request
API monolith → MongoDBRequest
db.messages.find({ channel_id: 1234 }).sort({ created_at: -1 }).limit(50)12
index: { channel_id: 1, created_at: -1 }
stored: about 100 000 000 messages (November 2015)
-- the data and the index no longer fit in RAM3
channel_id— Every chat read is "the latest messages of one channel".created_at— Sort order. Later replaced with Snowflake ids: two messages can share a timestamp.RAM— Once the index spills out of memory, each read may touch the disk.
Build it yourselfCheat sheetCompare lessons
Course: 74 lessons
01 Foundations
02 Scaling
03 Data & storage
04 Communication
05 Reliability
06 Architecture
07 Security
08 Shipping
09 Whole systems
10 Business processes
11 Case studies