Konspekt · Ishonchlilik
Backpressure
Muammo
Mobil ilova analitika hodisalarini ingest API'ga yuboradi, API ularni writer bazaga yozguncha o'z xotirasida saqlaydi. Baza sekinlashsa, hodisalar chiqib ketishidan tezroq keladi, buferda chegara yo'q, va API xotirasi tugab o'ladi: o'shanda 1,8 million hodisaga allaqachon "qabul qildim" deb javob berilgan bo'ladi. Backpressure tizimning zanjir bo'ylab orqaga "sekinroq" deyish usuli: chegarali bufer, to'lganda nima qilish qoidasi, Retry-After bilan 429, o'zi tortib oladigan consumer'lar, ustuvorlik bo'yicha rad etish va kirishdagi limit.
Nimani o'rganasiz
- Nega chegarasiz navbat ortiqcha yukni avval kechikishga, keyin yo'qolgan ma'lumotga aylantiradi va buni qanday hisoblash
- Chegarali bufer va to'lib qolganda to'rt yo'l: kutish (block), yangisini tashlash, eskisini tashlash, sampling
- 429 va Retry-After bilan yukni rad etish, pull asosidagi oqim: consumer poll, TCP oynasi, request(n)
- Signalni ustuvorlik bo'yicha yuqoriga uzatish va navbat o'rniga tez rad etadigan concurrency limit
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_apps(["Mobil ilovalar"])
n_api["Ingest API"]
n_writer("Writer'lar")
n_db[("Analitika bazasi")]
n_log[["Hodisalar logi"]]
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_logMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_apps as Mobil ilovalar
participant n_api as Ingest API
participant n_buffer as Xotiradagi bufer
participant n_writer as Writer'lar
participant n_log as Hodisalar logi
Note over n_apps,n_api: Hodisalar xotiraga tushadi, writer ularni bo'shatadi
n_apps->>n_api: POST /v1/events HTTP/1.1
n_api-->>n_apps: HTTP/1.1 202 Accepted
Note over n_api,n_buffer: Baza sekinlashadi, bufer o'sadi
n_api->>n_buffer: 19:10 buffer in 1 200/s out 800/s
Note over n_api,n_buffer: API 1,8 million tasdiqlangan hodisa bilan o'ladi
n_api->>n_buffer: 20:16:41 api-3 OOMKilled (limit 2 GiB)
Note over n_api,n_buffer: Buferga chegara qo'yiladi: to'lsa, kutish
n_api->>n_buffer: buffer = BoundedQueue(capacity=50_000)
Note over n_api,n_buffer: To'lib qolganda: kutish, tashlash yoki sampling
n_api->>n_buffer: buffer full (50 000): what happens to event 50 001
Note over n_apps,n_api: Yukni rad etish: Retry-After bilan tez #34;yo'q#34;
n_apps->>n_api: POST /v1/events HTTP/1.1
n_api-->>n_apps: HTTP/1.1 429 Too Many Requests
Note over n_writer,n_log: Mustahkam log va o'zi tortib oladigan writer'lar
n_writer->>n_log: writer-2 poll(max_records=500)
Note over n_api,n_log: Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi
n_api->>n_log: api admission consumer_lag=74 s (limit 60 s)
Note over n_apps,n_api: Kirishda concurrency limit
n_apps->>n_api: api concurrency limiter window 1 sD2
direction: right
n_apps: "Mobil ilovalar" {
shape: rectangle
}
n_api: "Ingest API" {
shape: rectangle
}
n_writer: "Writer'lar" {
shape: rectangle
style.multiple: true
}
n_db: "Analitika bazasi" {
shape: cylinder
}
n_log: "Hodisalar logi" {
shape: queue
}
n_apps -> n_api
n_writer -> n_db
n_api -> n_log
n_writer -> n_logPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Mobil ilovalar" as n_apps
participant "Ingest API" as n_api
queue "Xotiradagi bufer" as n_buffer
participant "Writer'lar" as n_writer
queue "Hodisalar logi" as n_log
note over n_apps, n_api : Hodisalar xotiraga tushadi, writer ularni bo'shatadi
n_apps -> n_api : POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 202 Accepted
note over n_api, n_buffer : Baza sekinlashadi, bufer o'sadi
n_api -> n_buffer : 19:10 buffer in 1 200/s out 800/s
note over n_api, n_buffer : API 1,8 million tasdiqlangan hodisa bilan o'ladi
n_api -> n_buffer : 20:16:41 api-3 OOMKilled (limit 2 GiB)
note over n_api, n_buffer : Buferga chegara qo'yiladi: to'lsa, kutish
n_api -> n_buffer : buffer = BoundedQueue(capacity=50_000)
note over n_api, n_buffer : To'lib qolganda: kutish, tashlash yoki sampling
n_api -> n_buffer : buffer full (50 000): what happens to event 50 001
note over n_apps, n_api : Yukni rad etish: Retry-After bilan tez "yo'q"
n_apps -> n_api : [qayta urinish] POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 429 Too Many Requests
note over n_writer, n_log : Mustahkam log va o'zi tortib oladigan writer'lar
n_writer -> n_log : writer-2 poll(max_records=500)
note over n_api, n_log : Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi
n_api -> n_log : api admission consumer_lag=74 s (limit 60 s)
note over n_apps, n_api : Kirishda concurrency limit
n_apps -> n_api : api concurrency limiter window 1 s
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Mobil ilovalar" as n_apps
component "Ingest API" as n_api
collections "Writer'lar" as n_writer
database "Analitika bazasi" as n_db
queue "Hodisalar logi" as n_log
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_log
@endumlXulosa
Backpressure uchun olti qoida.
1) Yuk 1 dan oshsa, navbat har soniyada (kirish − chiqish) qadar o'sadi: umid qilmang, hisoblang. 2) Har buferning chegarasi bo'lsin, u qabul qilinadigan kechikishga qarab tanlanadi. 3) To'lganda nima bo'lishini hal qiling: block, drop newest, drop oldest yoki sample, va tashlanganini sanang. 4) Tez 429 va Retry-After bilan rad eting; mijozlar ma'lumotni saqlab, jitter bilan qayta urinadi. 5) Consumer'lar o'zi tortib olsin, ortiqcha yuk mustahkam logda tursin, alert consumer lag'ga qo'yilsin. 6) Signalni ustuvorlik bo'yicha yuqoriga uzating va kirishda concurrency'ni cheklang, shunda rad etish arzon bo'ladi.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| API xotirasi | 40 MB | 60 MB |
| Hodisa kechikishi | 0,5 s | 1 s |
| Yo'qolgan hodisalar | 0 | 0 |
| Rad etilgan hodisalar | 0% | 0% |
Qadamlar
- Holat Hodisalar xotiraga tushadi, writer ularni bo'shatadi
Ilovaning SDK'si hodisalarni yig'ib, 50 tagacha bo'lgan paketlarda yuboradi.
- Muammo Baza sekinlashadi, bufer o'sadi
19:00 da tungi indeks qayta qurilishi boshlanadi va writer'lar soniyasiga 800 tagacha tushadi.
- Muammo API 1,8 million tasdiqlangan hodisa bilan o'ladi
20:16 da API konteyneri 2 GiB limitiga yetadi va o'chirib yuboriladi.
- Yechim Buferga chegara qo'yiladi: to'lsa, kutish
Endi buferda ko'pi bilan 50 000 ta hodisa turadi: 50 MB atrofida, hech qachon ko'p emas.
- Yechim To'lib qolganda: kutish, tashlash yoki sampling
Kutish yagona javob emas.
- Yechim Yukni rad etish: Retry-After bilan tez "yo'q"
Kutib turgan so'rovlar baribir ulanish va thread'ni band qiladi.
- Yechim Mustahkam log va o'zi tortib oladigan writer'lar
Xotiradagi bufer o'rniga diskda, uchta replikali hodisalar logi keladi.
- Yechim Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi
API har 5 soniyada consumer lag'ni o'qiydi.
- Yechim Kirishda concurrency limit
Oxirgi qo'riqchi API'ning o'zida.
- Xulosa Xulosa
Backpressure uchun olti qoida.
Atamalar
- Backpressure
- Iste'molchi ulgurmay qolganda producer'ni sekinlatish. Navbat chegaralanadi va cheksiz o'sish o'rniga producer kutadi.
- Chegarali navbat (bounded queue)
- Sig'imi oldindan belgilangan navbat. To'lganda nima qilish (kutish, yangisini yoki eskisini tashlash) aniq qoida bilan hal qilinadi, xotira esa cheksiz o'smaydi.
- Consumer lag
- Consumer logning oxiridan qancha orqada ekani, yozuvlarda yoki soniyalarda. Ortiqcha yukning eng to'g'ri signali: alert CPU'ga emas, shunga qo'yiladi.
- Retry-After
- 429 yoki 503 javobidagi sarlavha: mijoz necha soniyadan keyin qaytishi kerak. Mijoz shu vaqt ustiga tasodifiy jitter qo'shib kutadi, ma'lumotini esa saqlab qoladi.
- Admission control
- Kirishda qaysi so'rovni qabul qilish, qaysini darhol rad etishni hal qilish: bir vaqtdagi so'rovlar limiti va ustuvorliklar bo'yicha. Tez rad etish arzon, navbatda kutish qimmat.
- Pull asosidagi oqim
- Consumer keyingi ishni faqat oldingisini tugatgach o'zi so'raydi (poll, request(n), TCP oynasi). Sur'atni producer emas, qabul qiluvchi belgilaydi.