Architecture / 4 / 9
Change data capture (CDC)
An online shop keeps orders in SQL and has to mirror every change into a search index and a data warehouse. Writing to the database and to the event log from the same code loses events. Reading the database's own change log does not, as long as you handle ordering, restarts, the first snapshot and schema changes.
- 15 min
- Advanced
- 12 steps
- Orders missing in search
- 0
- Search staleness
- 1 s
- Retained WAL
- 2 GB
- Duplicate events
- 0
- request
- reply
- redelivery
- duplicate
- lost
Numbers are a model scenario, not measurements
The real request
Orders API → Event logRequest
PRODUCE orders-events
key1: 5521
{
"order_id": 5521,
"status": 2"paid",
"total": 129000
}
Response
OK partition 3, offset 881233
key— The order id. All events of one order go to the same partition.status— The new state the search index must show.offset— The event's position in the log. The broker has it now.
Build it yourselfCheat sheetCompare lessons
Course: 74 lessons
01 Foundations
02 Scaling
03 Data & storage
04 Communication
05 Reliability
06 Architecture
07 Security
08 Shipping
09 Whole systems
10 Business processes
11 Case studies