mufassall

Architecture / 4 / 4

Batch vs stream

An ad network counts clicks to bill advertisers and stop ads when the budget runs out. A nightly job is simple but a day late; a stream is fresh but has to deal with time, late events and replays.

  • 14 min
  • Advanced
  • 11 steps
Count lag
86 400 s
Budget overspend
0 $
Miscounted
0 %
Open in Lab
  • request
  • reply
  • redelivery
  • duplicate
  • lost

Numbers are a model scenario, not measurements

The real request

Nightly job → Stats DB

Request

INSERT OVERWRITE daily_clicks1
PARTITION (day = '2026-11-17')2
SELECT ad_id, count(*) AS clicks
FROM raw_clicks
WHERE day = '2026-11-17'
GROUP BY ad_id;3

Response

wrote 18400 rows in 312 s
  1. OVERWRITE — The job replaces the whole day. Running it twice gives the same result, not double counts.
  2. PARTITION — One partition per day. A rerun touches only that day.
  3. GROUP BY — One total per ad, over the full day. All data is already there, nothing is late.
Course: 42 lessons
  1. 01 Foundations

    1. The life of a request
    2. Latency and throughput
    3. Napkin math
  2. 02 Scaling

    1. Scaling up vs scaling out
    2. Load balancing
    3. Stateless servers
    4. Caching
    5. CDN and edge caching
  3. 03 Data & storage

    1. SQL and NoSQL
    2. Database indexes
    3. Storage engines: B-tree and LSM
    4. Database replication
    5. Sharding
    6. Consistent hashing
    7. Unique IDs
    8. CAP: when the link breaks
    9. Consensus and leader election
    10. Transactions
    11. File uploads
  4. 04 Communication

    1. API styles
    2. Message queues
    3. Pub/sub and event logs
    4. Real-time updates
    5. Reliable callbacks
  5. 05 Reliability

    1. Redundancy and failover
    2. Rate limiting
    3. Timeouts and retries
    4. Observability
  6. 06 Architecture

    1. Microservices
    2. Service discovery
    3. Sagas and the outbox
    4. Batch vs stream
  7. 07 Security

    1. Authentication
    2. Authorization
    3. Encryption and secrets
  8. 08 Shipping

    1. Testing
    2. CI/CD: from commit to production
  9. 09 Whole systems

    1. From requirements to design
    2. Design a URL shortener
    3. Design a news feed
    4. Design a chat
  10. 10 Business processes

    1. Payment gateway