mufassall

Data & storage / 5 / 11

Sharding

A chat service's messages table reaches 6 TB and one server can no longer hold it. We split it across shards, choose a shard key, meet a hot shard and a hot key, and add a fourth shard without moving everything.

  • 13 min
  • Intermediate
  • 10 steps
Query p95
1 400 ms
Hottest shard load
98 %
Shards per query
1
Largest shard
6 TB
Open in Lab
45%98%
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Chat API → Messages DB

Request

INSERT INTO messages (chat_id, sender_id, body)1
VALUES ('c_8812', 'u_42', 'see you at 7');

Response

INSERT 0 1
Time: 1380.5 ms2
  1. chat_id — The chat this message belongs to. Almost every query filters on it.
  2. Time — 1.4 seconds to insert one message. The server is out of room, not the query.
Course: 42 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Storage engines: B-tree and LSM
    4. Database replication
    5. Sharding
    6. Consistent hashing
    7. Unique IDs
    8. CAP: when the link breaks
    9. Consensus and leader election
    10. Transactions
    11. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Batch vs stream
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
  10. 10 Business processes

    1. Payment gateway