Cheat sheet · Architecture
Batch vs stream
The problem
An ad network counts clicks to bill advertisers and stop ads when the budget runs out. A nightly job is simple but a day late; a stream is fresh but has to deal with time, late events and replays.
What you learn
- What a nightly batch job buys you and what its latency costs
- Tumbling and sliding windows, and what each is for
- Late events, event time and watermarks
- At-least-once delivery plus an idempotent sink gives exactly-once results
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_app(["Apps"])
n_collector["Click collector"]
n_files[("Raw click files")]
n_nightly("Nightly job")
n_stats[("Stats DB")]
n_ads["Ad server"]
n_topic[["Click topic"]]
n_stream("Stream processor")
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream -.->|"retry"| n_filesMermaid: request sequence
sequenceDiagram
participant n_nightly as Nightly job
participant n_stats as Stats DB
participant n_ads as Ad server
participant n_collector as Click collector
participant n_topic as Click topic
participant n_stream as Stream processor
Note over n_nightly,n_stats: A nightly job counts yesterday
n_nightly->>n_stats: INSERT OVERWRITE daily_clicks
n_stats-->>n_nightly: wrote 18400 rows in 312 s
Note over n_ads,n_stats: The budget check reads last night's numbers
n_ads->>n_stats: SELECT spent, budget FROM ad_budgets
n_stats-->>n_ads: spent = 0.00 #124; budget = 50.00
Note over n_collector,n_topic: Stream: count clicks as they arrive
n_collector->>n_topic: produce topic=clicks key=ad_7
n_topic-->>n_collector: ack partition=5 offset=88120
Note over n_stream,n_stats: Tumbling windows: fixed one-minute buckets
n_stream->>n_stats: INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats-->>n_stream: INSERT 0 1
Note over n_stream,n_ads: Sliding windows: catching click fraud
n_stream->>n_ads: POST /blocklist HTTP/1.1
n_ads-->>n_stream: HTTP/1.1 204 No Content
Note over n_collector,n_topic: Clicks that arrive minutes late
n_collector->>n_topic: produce topic=clicks key=ad_7
Note over n_stream,n_stats: Watermarks: how long to wait for stragglers
n_stream->>n_stats: watermark = 10:05:40 (max event_time - 5 min)
Note over n_topic,n_stream: Crash and replay: clicks counted twice
n_topic->>n_stream: consumer restarted, resume at committed offset 88120
Note over n_stream,n_stats: An idempotent sink: replays change nothing
n_stream->>n_stats: INSERT INTO ad_minute
n_stats-->>n_stream: INSERT 0 0 -- replay ignoredD2
direction: right
n_app: "Apps" {
shape: rectangle
}
n_collector: "Click collector" {
shape: rectangle
}
n_files: "Raw click files" {
shape: stored_data
}
n_nightly: "Nightly job" {
shape: rectangle
style.multiple: true
}
n_stats: "Stats DB" {
shape: cylinder
}
n_ads: "Ad server" {
shape: rectangle
}
n_topic: "Click topic" {
shape: queue
}
n_stream: "Stream processor" {
shape: rectangle
style.multiple: true
}
n_app -> n_collector
n_collector -> n_files
n_files -> n_nightly
n_nightly -> n_stats
n_ads -> n_stats
n_collector -> n_topic
n_topic -> n_stream
n_stream -> n_stats
n_stream -> n_ads
n_stream -> n_files: "retry" {
style.stroke-dash: 3
}PlantUML: request sequence
@startuml
participant "Nightly job" as n_nightly
database "Stats DB" as n_stats
participant "Ad server" as n_ads
participant "Click collector" as n_collector
queue "Click topic" as n_topic
participant "Stream processor" as n_stream
note over n_nightly, n_stats : A nightly job counts yesterday
n_nightly -> n_stats : INSERT OVERWRITE daily_clicks
n_stats --> n_nightly : wrote 18400 rows in 312 s
note over n_ads, n_stats : The budget check reads last night's numbers
n_ads -> n_stats : SELECT spent, budget FROM ad_budgets
n_stats --> n_ads : spent = 0.00 | budget = 50.00
note over n_collector, n_topic : Stream: count clicks as they arrive
n_collector -> n_topic : produce topic=clicks key=ad_7
n_topic --> n_collector : ack partition=5 offset=88120
note over n_stream, n_stats : Tumbling windows: fixed one-minute buckets
n_stream -> n_stats : INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats --> n_stream : INSERT 0 1
note over n_stream, n_ads : Sliding windows: catching click fraud
n_stream -> n_ads : POST /blocklist HTTP/1.1
n_ads --> n_stream : HTTP/1.1 204 No Content
note over n_collector, n_topic : Clicks that arrive minutes late
n_collector -> n_topic : produce topic=clicks key=ad_7
note over n_stream, n_stats : Watermarks: how long to wait for stragglers
n_stream -> n_stats : watermark = 10:05:40 (max event_time - 5 min)
note over n_topic, n_stream : Crash and replay: clicks counted twice
n_topic -> n_stream : [duplicate] consumer restarted, resume at committed offset 88120
note over n_stream, n_stats : An idempotent sink: replays change nothing
n_stream -> n_stats : INSERT INTO ad_minute
n_stats --> n_stream : INSERT 0 0 -- replay ignored
@endumlPlantUML: components
@startuml
left to right direction
agent "Apps" as n_app
component "Click collector" as n_collector
storage "Raw click files" as n_files
collections "Nightly job" as n_nightly
database "Stats DB" as n_stats
component "Ad server" as n_ads
queue "Click topic" as n_topic
collections "Stream processor" as n_stream
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream ..> n_files : retry
@endumlRecap
Four rules for batch and stream.
1) Batch is simple, rerunnable and late; stream is fresh and harder. 2) Window by event time: tumbling for counts, sliding for bursts. 3) A watermark decides how long to wait for late events; later ones go to a side output. 4) Delivery is at least once, so make the sink idempotent: write full values keyed by window and offset.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Count lag | 86 400 s | 360 s |
| Budget overspend | 0 $ | 6 $ |
| Miscounted | 0% | 0% |
Steps
- Case A nightly job counts yesterday
Apps send clicks to a collector, which appends them to files.
- Problem The budget check reads last night's numbers
The ad server stops an ad when its daily budget of $50 is spent.
- Fix Stream: count clicks as they arrive
The collector now also writes each click to a topic.
- Case Tumbling windows: fixed one-minute buckets
The processor groups clicks into tumbling windows: 10:00-10:01, 10:01-10:02, back to back, no overlap.
- Case Sliding windows: catching click fraud
Fraud rule: more than 20 clicks from one device within any 60 seconds.
- Problem Clicks that arrive minutes late
A phone in the metro has no network.
- Fix Watermarks: how long to wait for stragglers
The processor tracks a watermark: the newest event time seen minus 5 minutes of allowed lateness.
- Problem Crash and replay: clicks counted twice
The processor wrote counts up to offset 89320 and crashed before saving its position.
- Fix An idempotent sink: replays change nothing
The processor now writes the full count of the window, not a delta, together with the offset it has reached.
- Case Keep the batch job for the final word
The stream is fast and almost right; it drives budgets and fraud blocks.
- Recap Recap
Four rules for batch and stream.
Terms
- Batch vs stream
- A batch job processes data in bulk on a schedule, such as nightly: simple and rerunnable, but late. A stream counts events in windows as they arrive: fresh, but harder.
- Watermark
- A marker that decides how long a stream processor waits for late events: the newest event time seen minus the allowed lateness. A window is final only once the watermark passes its end; later events go to a side output.