mufassall

Konspekt · Arxitektura

Batch va stream

14 daq · Yuqori · mufassall.dtwin.uz/uz/l/batch-stream/

← Darsni ochish

Muammo

Reklama tarmog'i reklama beruvchiga hisob chiqarish va byudjet tugaganda reklamani to'xtatish uchun kliklarni sanaydi. Tungi job oddiy, lekin bir kun kechikadi; stream yangi, lekin vaqt, kech kelgan hodisalar va takrorlar bilan kurashishi kerak.

Nimani o'rganasiz

  • Tungi batch job nima beradi va uning kechikishi nimaga tushadi
  • Tumbling va sliding oynalar, har biri nima uchun
  • Kech kelgan hodisalar, event time va watermark
  • At-least-once yetkazish va idempotent yozuv aynan bir martalik natija beradi
Yakuniy sahna
70%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_app(["Ilovalar"])
  n_collector["Klik yig'uvchi"]
  n_files[("Xom klik fayllari")]
  n_nightly("Tungi job")
  n_stats[("Statistika bazasi")]
  n_ads["Reklama serveri"]
  n_topic[["Klik topic'i"]]
  n_stream("Stream protsessor")
  n_app --> n_collector
  n_collector --> n_files
  n_files --> n_nightly
  n_nightly --> n_stats
  n_ads --> n_stats
  n_collector --> n_topic
  n_topic --> n_stream
  n_stream --> n_stats
  n_stream --> n_ads
  n_stream -.->|"qayta urinish"| n_files

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_nightly as Tungi job
  participant n_stats as Statistika bazasi
  participant n_ads as Reklama serveri
  participant n_collector as Klik yig'uvchi
  participant n_topic as Klik topic'i
  participant n_stream as Stream protsessor
  Note over n_nightly,n_stats: Tungi job kechagi kunni sanaydi
  n_nightly->>n_stats: INSERT OVERWRITE daily_clicks
  n_stats-->>n_nightly: wrote 18400 rows in 312 s
  Note over n_ads,n_stats: Byudjet tekshiruvi kechagi raqamni o'qiydi
  n_ads->>n_stats: SELECT spent, budget FROM ad_budgets
  n_stats-->>n_ads: spent = 0.00 #124; budget = 50.00
  Note over n_collector,n_topic: Stream: kliklarni kelishi bilan sanash
  n_collector->>n_topic: produce topic=clicks key=ad_7
  n_topic-->>n_collector: ack partition=5 offset=88120
  Note over n_stream,n_stats: Tumbling oynalar: qat'iy bir daqiqalik bo'laklar
  n_stream->>n_stats: INSERT INTO ad_minute (ad_id, window_start, clicks)
  n_stats-->>n_stream: INSERT 0 1
  Note over n_stream,n_ads: Sliding oynalar: klik firibgarligini ushlash
  n_stream->>n_ads: POST /blocklist HTTP/1.1
  n_ads-->>n_stream: HTTP/1.1 204 No Content
  Note over n_collector,n_topic: Daqiqalab kechikib keladigan kliklar
  n_collector->>n_topic: produce topic=clicks key=ad_7
  Note over n_stream,n_stats: Watermark: kechikkanlarni qancha kutish kerak
  n_stream->>n_stats: watermark = 10:05:40 (max event_time - 5 min)
  Note over n_topic,n_stream: Yiqilish va qayta o'qish: kliklar ikki marta sanaldi
  n_topic->>n_stream: consumer restarted, resume at committed offset 88120
  Note over n_stream,n_stats: Idempotent yozuv: takror hech narsani o'zgartirmaydi
  n_stream->>n_stats: INSERT INTO ad_minute
  n_stats-->>n_stream: INSERT 0 0 -- replay ignored

D2

direction: right

n_app: "Ilovalar" {
  shape: rectangle
}
n_collector: "Klik yig'uvchi" {
  shape: rectangle
}
n_files: "Xom klik fayllari" {
  shape: stored_data
}
n_nightly: "Tungi job" {
  shape: rectangle
  style.multiple: true
}
n_stats: "Statistika bazasi" {
  shape: cylinder
}
n_ads: "Reklama serveri" {
  shape: rectangle
}
n_topic: "Klik topic'i" {
  shape: queue
}
n_stream: "Stream protsessor" {
  shape: rectangle
  style.multiple: true
}

n_app -> n_collector
n_collector -> n_files
n_files -> n_nightly
n_nightly -> n_stats
n_ads -> n_stats
n_collector -> n_topic
n_topic -> n_stream
n_stream -> n_stats
n_stream -> n_ads
n_stream -> n_files: "qayta urinish" {
  style.stroke-dash: 3
}

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Tungi job" as n_nightly
database "Statistika bazasi" as n_stats
participant "Reklama serveri" as n_ads
participant "Klik yig'uvchi" as n_collector
queue "Klik topic'i" as n_topic
participant "Stream protsessor" as n_stream
note over n_nightly, n_stats : Tungi job kechagi kunni sanaydi
n_nightly -> n_stats : INSERT OVERWRITE daily_clicks
n_stats --> n_nightly : wrote 18400 rows in 312 s
note over n_ads, n_stats : Byudjet tekshiruvi kechagi raqamni o'qiydi
n_ads -> n_stats : SELECT spent, budget FROM ad_budgets
n_stats --> n_ads : spent = 0.00 | budget = 50.00
note over n_collector, n_topic : Stream: kliklarni kelishi bilan sanash
n_collector -> n_topic : produce topic=clicks key=ad_7
n_topic --> n_collector : ack partition=5 offset=88120
note over n_stream, n_stats : Tumbling oynalar: qat'iy bir daqiqalik bo'laklar
n_stream -> n_stats : INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats --> n_stream : INSERT 0 1
note over n_stream, n_ads : Sliding oynalar: klik firibgarligini ushlash
n_stream -> n_ads : POST /blocklist HTTP/1.1
n_ads --> n_stream : HTTP/1.1 204 No Content
note over n_collector, n_topic : Daqiqalab kechikib keladigan kliklar
n_collector -> n_topic : produce topic=clicks key=ad_7
note over n_stream, n_stats : Watermark: kechikkanlarni qancha kutish kerak
n_stream -> n_stats : watermark = 10:05:40 (max event_time - 5 min)
note over n_topic, n_stream : Yiqilish va qayta o'qish: kliklar ikki marta sanaldi
n_topic -> n_stream : [dublikat] consumer restarted, resume at committed offset 88120
note over n_stream, n_stats : Idempotent yozuv: takror hech narsani o'zgartirmaydi
n_stream -> n_stats : INSERT INTO ad_minute
n_stats --> n_stream : INSERT 0 0 -- replay ignored
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Ilovalar" as n_app
component "Klik yig'uvchi" as n_collector
storage "Xom klik fayllari" as n_files
collections "Tungi job" as n_nightly
database "Statistika bazasi" as n_stats
component "Reklama serveri" as n_ads
queue "Klik topic'i" as n_topic
collections "Stream protsessor" as n_stream
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream ..> n_files : qayta urinish
@enduml

Xulosa

Batch va stream uchun to'rt qoida.

1) Batch oddiy, qayta ishga tushiriladi va kechikadi; stream yangi va murakkabroq. 2) Oynani event time bo'yicha tuzing: sanoq uchun tumbling, portlash uchun sliding. 3) Watermark kech hodisalarni qancha kutishni hal qiladi; undan kechroqlari side output'ga ketadi. 4) Yetkazish kamida bir marta, shuning uchun yozuvni idempotent qiling: oyna va offset bo'yicha to'liq qiymat yozing.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Sanoq kechikishi86 400 s360 s
Byudjetdan oshish0 $6 $
Noto'g'ri sanalgan0%0%

Qadamlar

  1. Holat Tungi job kechagi kunni sanaydi

    Ilovalar kliklarni yig'uvchiga yuboradi, u ularni fayllarga qo'shib boradi.

  2. Muammo Byudjet tekshiruvi kechagi raqamni o'qiydi

    Reklama serveri kunlik $50 byudjet tugaganda reklamani to'xtatadi.

  3. Yechim Stream: kliklarni kelishi bilan sanash

    Yig'uvchi endi har klikni topic'ga ham yozadi.

  4. Holat Tumbling oynalar: qat'iy bir daqiqalik bo'laklar

    Protsessor kliklarni tumbling oynalarga ajratadi: 10:00-10:01, 10:01-10:02, ketma-ket, ustma-ust tushmaydi.

  5. Holat Sliding oynalar: klik firibgarligini ushlash

    Firibgarlik qoidasi: istalgan 60 soniya ichida bitta qurilmadan 20 dan ortiq klik.

  6. Muammo Daqiqalab kechikib keladigan kliklar

    Metroda telefonda tarmoq yo'q.

  7. Yechim Watermark: kechikkanlarni qancha kutish kerak

    Protsessor watermark yuritadi: ko'rilgan eng yangi event time minus 5 daqiqa ruxsat etilgan kechikish.

  8. Muammo Yiqilish va qayta o'qish: kliklar ikki marta sanaldi

    Protsessor sanoqlarni 89320 offset'gacha yozdi va o'z joyini saqlashdan oldin yiqildi.

  9. Yechim Idempotent yozuv: takror hech narsani o'zgartirmaydi

    Protsessor endi delta emas, oynaning to'liq sanog'ini va yetib kelgan offset'ni yozadi.

  10. Holat Oxirgi so'z uchun batch job'ni qoldiring

    Stream tez va deyarli to'g'ri; u byudjetlar va firibgarlik bloklarini boshqaradi.

  11. Xulosa Xulosa

    Batch va stream uchun to'rt qoida.

Atamalar

Batch va stream
Batch job ma'lumotni davriy, masalan har kecha bir yo'la ishlaydi: oddiy va qayta yurgizsa bo'ladi, lekin kechikadi. Stream hodisalarni kelishi bilan oynalar bo'yicha sanaydi: yangi, lekin murakkabroq.
Watermark
Stream processor kech kelgan hodisalarni qancha kutishini hal qiladigan belgi: ko'rilgan eng yangi event time minus ruxsat etilgan kechikish. Oyna watermark uning oxiridan o'tgandagina yakuniy bo'ladi, undan kechroqlari side output'ga ketadi.