mufassall

Shipping / 3 / 4

Zero-downtime migrations

A shop wants customer phone numbers in one normalised format, in a new column of a 40-million-row table, without stopping checkout. A plain rename breaks the pods that still run the old code, and a 5 ms ALTER stops every query because it waits for a lock behind a long report. The safe way is expand and contract: add a nullable column, write both, backfill in batches, verify and switch reads, stop the old writes, and drop the old column last, with a way back at every phase.

  • 15 min
  • Advanced
  • 10 steps
Checkout errors
0.1 %
Longest blocked query
0 s
Rows backfilled
0 %
Time to roll back
15 min
Open in Lab

50%
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Migrations → Customers DB

Request

-- migration 0042, runs right before the rollout1
ALTER TABLE customers
  RENAME COLUMN phone TO phone_e164;23
-- then deploy v2, which reads phone_e164
-- customers: 40 M rows, checkout writes 900 rows/s4

Response

ALTER TABLE  12 ms
  1. 0042 — The migration runs in the pipeline, just before the new code.
  2. RENAME COLUMN — Instant: it only changes the name in the catalog.
  3. phone_e164 — The name the new code expects.
  4. 40 M rows — A big, hot table: every lock on it is felt at checkout.

Build it yourselfCheat sheetCompare lessons

Course: 74 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. Cache invalidation
    6. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Geo search: who is nearby
    4. Full-text search
    5. Vector search
    6. Storage engines: B-tree and LSM
    7. Bloom filters
    8. Database replication
    9. Sharding
    10. Consistent hashing
    11. Unique IDs
    12. CAP: when the link breaks
    13. Consensus and leader election
    14. Transactions
    15. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
    6. Sending webhooks reliably
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
    5. SLOs and error budgets
    6. Backpressure
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Change data capture (CDC)
    5. Transactional outbox
    6. Distributed locks
    7. Batch vs stream
    8. Multi-tenancy
    9. Event sourcing
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
    4. Secrets management
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
    3. Zero-downtime migrations
    4. Feature flags
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
    5. Design a video streaming site
    6. Design search autocomplete
    7. Design a job scheduler
  10. 10 Business processes

    1. Payment gateway
    2. Money and rounding
    3. Double-entry ledger
    4. Payment reconciliation
    5. Subscription billing
    6. Electronic digital signature
    7. Document lifecycle
    8. Audit log
    9. Notification delivery
    10. Bulk import
    11. Refunds and chargebacks
    12. Gapless document numbering
  11. 11 Case studies

    1. Case: how Discord stores messages
    2. Case: Figma multiplayer editing