Architecture / 4 / 4
Batch vs stream
An ad network counts clicks to bill advertisers and stop ads when the budget runs out. A nightly job is simple but a day late; a stream is fresh but has to deal with time, late events and replays.
- 14 min
- Advanced
- 11 steps
- Count lag
- 86 400 s
- Budget overspend
- 0 $
- Miscounted
- 0 %
- request
- reply
- redelivery
- duplicate
- lost
Numbers are a model scenario, not measurements
The real request
Nightly job → Stats DBRequest
INSERT OVERWRITE daily_clicks1
PARTITION (day = '2026-11-17')2
SELECT ad_id, count(*) AS clicks
FROM raw_clicks
WHERE day = '2026-11-17'
GROUP BY ad_id;3
Response
wrote 18400 rows in 312 s
OVERWRITE— The job replaces the whole day. Running it twice gives the same result, not double counts.PARTITION— One partition per day. A rerun touches only that day.GROUP BY— One total per ad, over the full day. All data is already there, nothing is late.
Course: 42 lessons
01 Foundations
02 Scaling
03 Data & storage
04 Communication
05 Reliability
06 Architecture
07 Security
08 Shipping
09 Whole systems
10 Business processes