mufassall

Data & storage / 7 / 11

Unique IDs

An orders service gets its ids from a database counter. Then the data is split across two databases, and the counters start handing out the same numbers. How to make ids that are unique everywhere, sort well in an index and survive a clock that jumps back.

  • 12 min
  • Intermediate
  • 10 steps
Insert p95
4 ms
Duplicate ids
0 /day
Index page splits
2 /s
Open in Lab
35%
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Orders API → Shard 1

Request

INSERT INTO orders (user_id, total)
VALUES ('u_42', 125000)
RETURNING id;1

Response

id = 482132
  1. RETURNING — The database tells the API which id it just assigned. The API never picks the number itself.
  2. 48213 — The next value of the counter. An 8-byte integer, sorted in insert order.
Course: 42 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Storage engines: B-tree and LSM
    4. Database replication
    5. Sharding
    6. Consistent hashing
    7. Unique IDs
    8. CAP: when the link breaks
    9. Consensus and leader election
    10. Transactions
    11. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Batch vs stream
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
  10. 10 Business processes

    1. Payment gateway