Konspekt · Arxitektura
Batch va stream
Muammo
Reklama tarmog'i reklama beruvchiga hisob chiqarish va byudjet tugaganda reklamani to'xtatish uchun kliklarni sanaydi. Tungi job oddiy, lekin bir kun kechikadi; stream yangi, lekin vaqt, kech kelgan hodisalar va takrorlar bilan kurashishi kerak.
Nimani o'rganasiz
- Tungi batch job nima beradi va uning kechikishi nimaga tushadi
- Tumbling va sliding oynalar, har biri nima uchun
- Kech kelgan hodisalar, event time va watermark
- At-least-once yetkazish va idempotent yozuv aynan bir martalik natija beradi
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_app(["Ilovalar"])
n_collector["Klik yig'uvchi"]
n_files[("Xom klik fayllari")]
n_nightly("Tungi job")
n_stats[("Statistika bazasi")]
n_ads["Reklama serveri"]
n_topic[["Klik topic'i"]]
n_stream("Stream protsessor")
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream -.->|"qayta urinish"| n_filesMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_nightly as Tungi job
participant n_stats as Statistika bazasi
participant n_ads as Reklama serveri
participant n_collector as Klik yig'uvchi
participant n_topic as Klik topic'i
participant n_stream as Stream protsessor
Note over n_nightly,n_stats: Tungi job kechagi kunni sanaydi
n_nightly->>n_stats: INSERT OVERWRITE daily_clicks
n_stats-->>n_nightly: wrote 18400 rows in 312 s
Note over n_ads,n_stats: Byudjet tekshiruvi kechagi raqamni o'qiydi
n_ads->>n_stats: SELECT spent, budget FROM ad_budgets
n_stats-->>n_ads: spent = 0.00 #124; budget = 50.00
Note over n_collector,n_topic: Stream: kliklarni kelishi bilan sanash
n_collector->>n_topic: produce topic=clicks key=ad_7
n_topic-->>n_collector: ack partition=5 offset=88120
Note over n_stream,n_stats: Tumbling oynalar: qat'iy bir daqiqalik bo'laklar
n_stream->>n_stats: INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats-->>n_stream: INSERT 0 1
Note over n_stream,n_ads: Sliding oynalar: klik firibgarligini ushlash
n_stream->>n_ads: POST /blocklist HTTP/1.1
n_ads-->>n_stream: HTTP/1.1 204 No Content
Note over n_collector,n_topic: Daqiqalab kechikib keladigan kliklar
n_collector->>n_topic: produce topic=clicks key=ad_7
Note over n_stream,n_stats: Watermark: kechikkanlarni qancha kutish kerak
n_stream->>n_stats: watermark = 10:05:40 (max event_time - 5 min)
Note over n_topic,n_stream: Yiqilish va qayta o'qish: kliklar ikki marta sanaldi
n_topic->>n_stream: consumer restarted, resume at committed offset 88120
Note over n_stream,n_stats: Idempotent yozuv: takror hech narsani o'zgartirmaydi
n_stream->>n_stats: INSERT INTO ad_minute
n_stats-->>n_stream: INSERT 0 0 -- replay ignoredD2
direction: right
n_app: "Ilovalar" {
shape: rectangle
}
n_collector: "Klik yig'uvchi" {
shape: rectangle
}
n_files: "Xom klik fayllari" {
shape: stored_data
}
n_nightly: "Tungi job" {
shape: rectangle
style.multiple: true
}
n_stats: "Statistika bazasi" {
shape: cylinder
}
n_ads: "Reklama serveri" {
shape: rectangle
}
n_topic: "Klik topic'i" {
shape: queue
}
n_stream: "Stream protsessor" {
shape: rectangle
style.multiple: true
}
n_app -> n_collector
n_collector -> n_files
n_files -> n_nightly
n_nightly -> n_stats
n_ads -> n_stats
n_collector -> n_topic
n_topic -> n_stream
n_stream -> n_stats
n_stream -> n_ads
n_stream -> n_files: "qayta urinish" {
style.stroke-dash: 3
}PlantUML: so'rovlar ketma-ketligi
@startuml
participant "Tungi job" as n_nightly
database "Statistika bazasi" as n_stats
participant "Reklama serveri" as n_ads
participant "Klik yig'uvchi" as n_collector
queue "Klik topic'i" as n_topic
participant "Stream protsessor" as n_stream
note over n_nightly, n_stats : Tungi job kechagi kunni sanaydi
n_nightly -> n_stats : INSERT OVERWRITE daily_clicks
n_stats --> n_nightly : wrote 18400 rows in 312 s
note over n_ads, n_stats : Byudjet tekshiruvi kechagi raqamni o'qiydi
n_ads -> n_stats : SELECT spent, budget FROM ad_budgets
n_stats --> n_ads : spent = 0.00 | budget = 50.00
note over n_collector, n_topic : Stream: kliklarni kelishi bilan sanash
n_collector -> n_topic : produce topic=clicks key=ad_7
n_topic --> n_collector : ack partition=5 offset=88120
note over n_stream, n_stats : Tumbling oynalar: qat'iy bir daqiqalik bo'laklar
n_stream -> n_stats : INSERT INTO ad_minute (ad_id, window_start, clicks)
n_stats --> n_stream : INSERT 0 1
note over n_stream, n_ads : Sliding oynalar: klik firibgarligini ushlash
n_stream -> n_ads : POST /blocklist HTTP/1.1
n_ads --> n_stream : HTTP/1.1 204 No Content
note over n_collector, n_topic : Daqiqalab kechikib keladigan kliklar
n_collector -> n_topic : produce topic=clicks key=ad_7
note over n_stream, n_stats : Watermark: kechikkanlarni qancha kutish kerak
n_stream -> n_stats : watermark = 10:05:40 (max event_time - 5 min)
note over n_topic, n_stream : Yiqilish va qayta o'qish: kliklar ikki marta sanaldi
n_topic -> n_stream : [dublikat] consumer restarted, resume at committed offset 88120
note over n_stream, n_stats : Idempotent yozuv: takror hech narsani o'zgartirmaydi
n_stream -> n_stats : INSERT INTO ad_minute
n_stats --> n_stream : INSERT 0 0 -- replay ignored
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Ilovalar" as n_app
component "Klik yig'uvchi" as n_collector
storage "Xom klik fayllari" as n_files
collections "Tungi job" as n_nightly
database "Statistika bazasi" as n_stats
component "Reklama serveri" as n_ads
queue "Klik topic'i" as n_topic
collections "Stream protsessor" as n_stream
n_app --> n_collector
n_collector --> n_files
n_files --> n_nightly
n_nightly --> n_stats
n_ads --> n_stats
n_collector --> n_topic
n_topic --> n_stream
n_stream --> n_stats
n_stream --> n_ads
n_stream ..> n_files : qayta urinish
@endumlXulosa
Batch va stream uchun to'rt qoida.
1) Batch oddiy, qayta ishga tushiriladi va kechikadi; stream yangi va murakkabroq. 2) Oynani event time bo'yicha tuzing: sanoq uchun tumbling, portlash uchun sliding. 3) Watermark kech hodisalarni qancha kutishni hal qiladi; undan kechroqlari side output'ga ketadi. 4) Yetkazish kamida bir marta, shuning uchun yozuvni idempotent qiling: oyna va offset bo'yicha to'liq qiymat yozing.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Sanoq kechikishi | 86 400 s | 360 s |
| Byudjetdan oshish | 0 $ | 6 $ |
| Noto'g'ri sanalgan | 0% | 0% |
Qadamlar
- Holat Tungi job kechagi kunni sanaydi
Ilovalar kliklarni yig'uvchiga yuboradi, u ularni fayllarga qo'shib boradi.
- Muammo Byudjet tekshiruvi kechagi raqamni o'qiydi
Reklama serveri kunlik $50 byudjet tugaganda reklamani to'xtatadi.
- Yechim Stream: kliklarni kelishi bilan sanash
Yig'uvchi endi har klikni topic'ga ham yozadi.
- Holat Tumbling oynalar: qat'iy bir daqiqalik bo'laklar
Protsessor kliklarni tumbling oynalarga ajratadi: 10:00-10:01, 10:01-10:02, ketma-ket, ustma-ust tushmaydi.
- Holat Sliding oynalar: klik firibgarligini ushlash
Firibgarlik qoidasi: istalgan 60 soniya ichida bitta qurilmadan 20 dan ortiq klik.
- Muammo Daqiqalab kechikib keladigan kliklar
Metroda telefonda tarmoq yo'q.
- Yechim Watermark: kechikkanlarni qancha kutish kerak
Protsessor watermark yuritadi: ko'rilgan eng yangi event time minus 5 daqiqa ruxsat etilgan kechikish.
- Muammo Yiqilish va qayta o'qish: kliklar ikki marta sanaldi
Protsessor sanoqlarni 89320 offset'gacha yozdi va o'z joyini saqlashdan oldin yiqildi.
- Yechim Idempotent yozuv: takror hech narsani o'zgartirmaydi
Protsessor endi delta emas, oynaning to'liq sanog'ini va yetib kelgan offset'ni yozadi.
- Holat Oxirgi so'z uchun batch job'ni qoldiring
Stream tez va deyarli to'g'ri; u byudjetlar va firibgarlik bloklarini boshqaradi.
- Xulosa Xulosa
Batch va stream uchun to'rt qoida.
Atamalar
- Batch va stream
- Batch job ma'lumotni davriy, masalan har kecha bir yo'la ishlaydi: oddiy va qayta yurgizsa bo'ladi, lekin kechikadi. Stream hodisalarni kelishi bilan oynalar bo'yicha sanaydi: yangi, lekin murakkabroq.
- Watermark
- Stream processor kech kelgan hodisalarni qancha kutishini hal qiladigan belgi: ko'rilgan eng yangi event time minus ruxsat etilgan kechikish. Oyna watermark uning oxiridan o'tgandagina yakuniy bo'ladi, undan kechroqlari side output'ga ketadi.