mufassall

Cheat sheet · Architecture

Batch vs stream

14 min · Advanced · mufassall.dtwin.uz/en/l/batch-stream/

← Open the lesson

The problem

An ad network counts clicks to bill advertisers and stop ads when the budget runs out. A nightly job is simple but a day late; a stream is fresh but has to deal with time, late events and replays.

What you learn

  • What a nightly batch job buys you and what its latency costs
  • Tumbling and sliding windows, and what each is for
  • Late events, event time and watermarks
  • At-least-once delivery plus an idempotent sink gives exactly-once results
Final scene
70%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_app(["Apps"])
  n_collector["Click collector"]
  n_files[("Raw click files")]
  n_nightly("Nightly job")
  n_stats[("Stats DB")]
  n_ads["Ad server"]
  n_topic[["Click topic"]]
  n_stream("Stream processor")
  n_app --> n_collector
  n_collector --> n_files
  n_files --> n_nightly
  n_nightly --> n_stats
  n_ads --> n_stats
  n_collector --> n_topic
  n_topic --> n_stream
  n_stream --> n_stats
  n_stream --> n_ads
  n_stream -.->|"retry"| n_files

Mermaid: request sequence

sequenceDiagram
  participant n_nightly as Nightly job
  participant n_stats as Stats DB
  participant n_ads as Ad server
  participant n_collector as Click collector
  participant n_topic as Click topic
  participant n_stream as Stream processor
  Note over n_nightly,n_stats: A nightly job counts yesterday
  n_nightly->>n_stats: INSERT OVERWRITE daily_clicks
  n_stats-->>n_nightly: wrote 18400 rows in 312 s
  Note over n_ads,n_stats: The budget check reads last night's numbers
  n_ads->>n_stats: SELECT spent, budget FROM ad_budgets
  n_stats-->>n_ads: spent = 0.00 #124; budget = 50.00
  Note over n_collector,n_topic: Stream: count clicks as they arrive
  n_collector->>n_topic: produce topic=clicks key=ad_7
  n_topic-->>n_collector: ack partition=5 offset=88120
  Note over n_stream,n_stats: Tumbling windows: fixed one-minute buckets
  n_stream->>n_stats: INSERT INTO ad_minute (ad_id, window_start, clicks)
  n_stats-->>n_stream: INSERT 0 1
  Note over n_stream,n_ads: Sliding windows: catching click fraud
  n_stream->>n_ads: POST /blocklist HTTP/1.1
  n_ads-->>n_stream: HTTP/1.1 204 No Content
  Note over n_collector,n_topic: Clicks that arrive minutes late
  n_collector->>n_topic: produce topic=clicks key=ad_7
  Note over n_stream,n_stats: Watermarks: how long to wait for stragglers
  n_stream->>n_stats: watermark = 10:05:40 (max event_time - 5 min)
  Note over n_topic,n_stream: Crash and replay: clicks counted twice
  n_topic->>n_stream: consumer restarted, resume at committed offset 88120
  Note over n_stream,n_stats: An idempotent sink: replays change nothing
  n_stream->>n_stats: INSERT INTO ad_minute
  n_stats-->>n_stream: INSERT 0 0 -- replay ignored

D2

direction: right

n_app: "Apps" {
  shape: rectangle
}
n_collector: "Click collector" {
  shape: rectangle
}
n_files: "Raw click files" {
  shape: stored_data
}
n_nightly: "Nightly job" {
  shape: rectangle
  style.multiple: true
}
n_stats: "Stats DB" {
  shape: cylinder
}
n_ads: "Ad server" {
  shape: rectangle
}
n_topic: "Click topic" {
  shape: queue
}
n_stream: "Stream processor" {
  shape: rectangle
  style.multiple: true
}

n_app -> n_collector
n_collector -> n_files
n_files -> n_nightly
n_nightly -> n_stats
n_ads -> n_stats
n_collector -> n_topic
n_topic -> n_stream
n_stream -> n_stats
n_stream -> n_ads
n_stream -> n_files: "retry" {
  style.stroke-dash: 3
}

PlantUML: request sequence

@startuml
participant "Nightly job" as n_nightly
database "Stats DB" as n_stats
participant "Ad server" as n_ads
participant "Click collector" as n_collector
queue "Click topic" as n_topic
participant "Stream processor" as n_stream
note over n_nightly, n_stats : A nightly job counts yesterday
n_nightly -> n_stats : INSERT OVERWRITE daily_clicks
n_stats --> n_nightly : wrote 18400 rows in 312 s
note over n_ads, n_stats : The budget check reads last night's numbers
n_ads -> n_stats : SELECT spent, budget FROM ad_budgets
n_stats --> n_ads : spent = 0.00 | budget = 50.00
note over n_collector, n_topic : Stream: count clicks as they arrive
n_collector -> n_topic : produce topic=clicks key=ad_7
n_topic --> n_collector : ack partition=5 offset=88120
note over n_stream, n_stats : Tumbling windows: fixed one-minute buckets
n_stream -> n_stats : INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats --> n_stream : INSERT 0 1
note over n_stream, n_ads : Sliding windows: catching click fraud
n_stream -> n_ads : POST /blocklist HTTP/1.1
n_ads --> n_stream : HTTP/1.1 204 No Content
note over n_collector, n_topic : Clicks that arrive minutes late
n_collector -> n_topic : produce topic=clicks key=ad_7
note over n_stream, n_stats : Watermarks: how long to wait for stragglers
n_stream -> n_stats : watermark = 10:05:40 (max event_time - 5 min)
note over n_topic, n_stream : Crash and replay: clicks counted twice
n_topic -> n_stream : [duplicate] consumer restarted, resume at committed offset 88120
note over n_stream, n_stats : An idempotent sink: replays change nothing
n_stream -> n_stats : INSERT INTO ad_minute
n_stats --> n_stream : INSERT 0 0 -- replay ignored
@enduml

PlantUML: components

@startuml
left to right direction
agent "Apps" as n_app
component "Click collector" as n_collector
storage "Raw click files" as n_files
collections "Nightly job" as n_nightly
database "Stats DB" as n_stats
component "Ad server" as n_ads
queue "Click topic" as n_topic
collections "Stream processor" as n_stream
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream ..> n_files : retry
@enduml

Recap

Four rules for batch and stream.

1) Batch is simple, rerunnable and late; stream is fresh and harder. 2) Window by event time: tumbling for counts, sliding for bursts. 3) A watermark decides how long to wait for late events; later ones go to a side output. 4) Delivery is at least once, so make the sink idempotent: write full values keyed by window and offset.

Numbers: at the start and at the end

MetricStartEnd
Count lag86 400 s360 s
Budget overspend0 $6 $
Miscounted0%0%

Steps

  1. Case A nightly job counts yesterday

    Apps send clicks to a collector, which appends them to files.

  2. Problem The budget check reads last night's numbers

    The ad server stops an ad when its daily budget of $50 is spent.

  3. Fix Stream: count clicks as they arrive

    The collector now also writes each click to a topic.

  4. Case Tumbling windows: fixed one-minute buckets

    The processor groups clicks into tumbling windows: 10:00-10:01, 10:01-10:02, back to back, no overlap.

  5. Case Sliding windows: catching click fraud

    Fraud rule: more than 20 clicks from one device within any 60 seconds.

  6. Problem Clicks that arrive minutes late

    A phone in the metro has no network.

  7. Fix Watermarks: how long to wait for stragglers

    The processor tracks a watermark: the newest event time seen minus 5 minutes of allowed lateness.

  8. Problem Crash and replay: clicks counted twice

    The processor wrote counts up to offset 89320 and crashed before saving its position.

  9. Fix An idempotent sink: replays change nothing

    The processor now writes the full count of the window, not a delta, together with the offset it has reached.

  10. Case Keep the batch job for the final word

    The stream is fast and almost right; it drives budgets and fraud blocks.

  11. Recap Recap

    Four rules for batch and stream.

Terms

Batch vs stream
A batch job processes data in bulk on a schedule, such as nightly: simple and rerunnable, but late. A stream counts events in windows as they arrive: fresh, but harder.
Watermark
A marker that decides how long a stream processor waits for late events: the newest event time seen minus the allowed lateness. A window is final only once the watermark passes its end; later events go to a side output.