mufassall

Konspekt · Ishonchlilik

Backpressure

14 daq · O'rta · mufassall.dtwin.uz/uz/l/backpressure/

← Darsni ochish

Muammo

Mobil ilova analitika hodisalarini ingest API'ga yuboradi, API ularni writer bazaga yozguncha o'z xotirasida saqlaydi. Baza sekinlashsa, hodisalar chiqib ketishidan tezroq keladi, buferda chegara yo'q, va API xotirasi tugab o'ladi: o'shanda 1,8 million hodisaga allaqachon "qabul qildim" deb javob berilgan bo'ladi. Backpressure tizimning zanjir bo'ylab orqaga "sekinroq" deyish usuli: chegarali bufer, to'lganda nima qilish qoidasi, Retry-After bilan 429, o'zi tortib oladigan consumer'lar, ustuvorlik bo'yicha rad etish va kirishdagi limit.

Nimani o'rganasiz

  • Nega chegarasiz navbat ortiqcha yukni avval kechikishga, keyin yo'qolgan ma'lumotga aylantiradi va buni qanday hisoblash
  • Chegarali bufer va to'lib qolganda to'rt yo'l: kutish (block), yangisini tashlash, eskisini tashlash, sampling
  • 429 va Retry-After bilan yukni rad etish, pull asosidagi oqim: consumer poll, TCP oynasi, request(n)
  • Signalni ustuvorlik bo'yicha yuqoriga uzatish va navbat o'rniga tez rad etadigan concurrency limit
Yakuniy sahna
60%70%20%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_apps(["Mobil ilovalar"])
  n_api["Ingest API"]
  n_writer("Writer'lar")
  n_db[("Analitika bazasi")]
  n_log[["Hodisalar logi"]]
  n_apps --> n_api
  n_writer --> n_db
  n_api --> n_log
  n_writer --> n_log

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_apps as Mobil ilovalar
  participant n_api as Ingest API
  participant n_buffer as Xotiradagi bufer
  participant n_writer as Writer'lar
  participant n_log as Hodisalar logi
  Note over n_apps,n_api: Hodisalar xotiraga tushadi, writer ularni bo'shatadi
  n_apps->>n_api: POST /v1/events HTTP/1.1
  n_api-->>n_apps: HTTP/1.1 202 Accepted
  Note over n_api,n_buffer: Baza sekinlashadi, bufer o'sadi
  n_api->>n_buffer: 19:10 buffer in 1 200/s out 800/s
  Note over n_api,n_buffer: API 1,8 million tasdiqlangan hodisa bilan o'ladi
  n_api->>n_buffer: 20:16:41 api-3 OOMKilled (limit 2 GiB)
  Note over n_api,n_buffer: Buferga chegara qo'yiladi: to'lsa, kutish
  n_api->>n_buffer: buffer = BoundedQueue(capacity=50_000)
  Note over n_api,n_buffer: To'lib qolganda: kutish, tashlash yoki sampling
  n_api->>n_buffer: buffer full (50 000): what happens to event 50 001
  Note over n_apps,n_api: Yukni rad etish: Retry-After bilan tez #34;yo'q#34;
  n_apps->>n_api: POST /v1/events HTTP/1.1
  n_api-->>n_apps: HTTP/1.1 429 Too Many Requests
  Note over n_writer,n_log: Mustahkam log va o'zi tortib oladigan writer'lar
  n_writer->>n_log: writer-2 poll(max_records=500)
  Note over n_api,n_log: Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi
  n_api->>n_log: api admission consumer_lag=74 s (limit 60 s)
  Note over n_apps,n_api: Kirishda concurrency limit
  n_apps->>n_api: api concurrency limiter window 1 s

D2

direction: right

n_apps: "Mobil ilovalar" {
  shape: rectangle
}
n_api: "Ingest API" {
  shape: rectangle
}
n_writer: "Writer'lar" {
  shape: rectangle
  style.multiple: true
}
n_db: "Analitika bazasi" {
  shape: cylinder
}
n_log: "Hodisalar logi" {
  shape: queue
}

n_apps -> n_api
n_writer -> n_db
n_api -> n_log
n_writer -> n_log

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Mobil ilovalar" as n_apps
participant "Ingest API" as n_api
queue "Xotiradagi bufer" as n_buffer
participant "Writer'lar" as n_writer
queue "Hodisalar logi" as n_log
note over n_apps, n_api : Hodisalar xotiraga tushadi, writer ularni bo'shatadi
n_apps -> n_api : POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 202 Accepted
note over n_api, n_buffer : Baza sekinlashadi, bufer o'sadi
n_api -> n_buffer : 19:10 buffer in 1 200/s out 800/s
note over n_api, n_buffer : API 1,8 million tasdiqlangan hodisa bilan o'ladi
n_api -> n_buffer : 20:16:41 api-3 OOMKilled (limit 2 GiB)
note over n_api, n_buffer : Buferga chegara qo'yiladi: to'lsa, kutish
n_api -> n_buffer : buffer = BoundedQueue(capacity=50_000)
note over n_api, n_buffer : To'lib qolganda: kutish, tashlash yoki sampling
n_api -> n_buffer : buffer full (50 000): what happens to event 50 001
note over n_apps, n_api : Yukni rad etish: Retry-After bilan tez "yo'q"
n_apps -> n_api : [qayta urinish] POST /v1/events HTTP/1.1
n_api --> n_apps : HTTP/1.1 429 Too Many Requests
note over n_writer, n_log : Mustahkam log va o'zi tortib oladigan writer'lar
n_writer -> n_log : writer-2 poll(max_records=500)
note over n_api, n_log : Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi
n_api -> n_log : api admission consumer_lag=74 s (limit 60 s)
note over n_apps, n_api : Kirishda concurrency limit
n_apps -> n_api : api concurrency limiter window 1 s
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Mobil ilovalar" as n_apps
component "Ingest API" as n_api
collections "Writer'lar" as n_writer
database "Analitika bazasi" as n_db
queue "Hodisalar logi" as n_log
n_apps --> n_api
n_writer --> n_db
n_api --> n_log
n_writer --> n_log
@enduml

Xulosa

Backpressure uchun olti qoida.

1) Yuk 1 dan oshsa, navbat har soniyada (kirish − chiqish) qadar o'sadi: umid qilmang, hisoblang. 2) Har buferning chegarasi bo'lsin, u qabul qilinadigan kechikishga qarab tanlanadi. 3) To'lganda nima bo'lishini hal qiling: block, drop newest, drop oldest yoki sample, va tashlanganini sanang. 4) Tez 429 va Retry-After bilan rad eting; mijozlar ma'lumotni saqlab, jitter bilan qayta urinadi. 5) Consumer'lar o'zi tortib olsin, ortiqcha yuk mustahkam logda tursin, alert consumer lag'ga qo'yilsin. 6) Signalni ustuvorlik bo'yicha yuqoriga uzating va kirishda concurrency'ni cheklang, shunda rad etish arzon bo'ladi.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
API xotirasi40 MB60 MB
Hodisa kechikishi0,5 s1 s
Yo'qolgan hodisalar00
Rad etilgan hodisalar0%0%

Qadamlar

  1. Holat Hodisalar xotiraga tushadi, writer ularni bo'shatadi

    Ilovaning SDK'si hodisalarni yig'ib, 50 tagacha bo'lgan paketlarda yuboradi.

  2. Muammo Baza sekinlashadi, bufer o'sadi

    19:00 da tungi indeks qayta qurilishi boshlanadi va writer'lar soniyasiga 800 tagacha tushadi.

  3. Muammo API 1,8 million tasdiqlangan hodisa bilan o'ladi

    20:16 da API konteyneri 2 GiB limitiga yetadi va o'chirib yuboriladi.

  4. Yechim Buferga chegara qo'yiladi: to'lsa, kutish

    Endi buferda ko'pi bilan 50 000 ta hodisa turadi: 50 MB atrofida, hech qachon ko'p emas.

  5. Yechim To'lib qolganda: kutish, tashlash yoki sampling

    Kutish yagona javob emas.

  6. Yechim Yukni rad etish: Retry-After bilan tez "yo'q"

    Kutib turgan so'rovlar baribir ulanish va thread'ni band qiladi.

  7. Yechim Mustahkam log va o'zi tortib oladigan writer'lar

    Xotiradagi bufer o'rniga diskda, uchta replikali hodisalar logi keladi.

  8. Yechim Kechikish zanjir bo'ylab ustuvorlik bilan yuqoriga chiqadi

    API har 5 soniyada consumer lag'ni o'qiydi.

  9. Yechim Kirishda concurrency limit

    Oxirgi qo'riqchi API'ning o'zida.

  10. Xulosa Xulosa

    Backpressure uchun olti qoida.

Atamalar

Backpressure
Iste'molchi ulgurmay qolganda producer'ni sekinlatish. Navbat chegaralanadi va cheksiz o'sish o'rniga producer kutadi.
Chegarali navbat (bounded queue)
Sig'imi oldindan belgilangan navbat. To'lganda nima qilish (kutish, yangisini yoki eskisini tashlash) aniq qoida bilan hal qilinadi, xotira esa cheksiz o'smaydi.
Consumer lag
Consumer logning oxiridan qancha orqada ekani, yozuvlarda yoki soniyalarda. Ortiqcha yukning eng to'g'ri signali: alert CPU'ga emas, shunga qo'yiladi.
Retry-After
429 yoki 503 javobidagi sarlavha: mijoz necha soniyadan keyin qaytishi kerak. Mijoz shu vaqt ustiga tasodifiy jitter qo'shib kutadi, ma'lumotini esa saqlab qoladi.
Admission control
Kirishda qaysi so'rovni qabul qilish, qaysini darhol rad etishni hal qilish: bir vaqtdagi so'rovlar limiti va ustuvorliklar bo'yicha. Tez rad etish arzon, navbatda kutish qimmat.
Pull asosidagi oqim
Consumer keyingi ishni faqat oldingisini tugatgach o'zi so'raydi (poll, request(n), TCP oynasi). Sur'atni producer emas, qabul qiluvchi belgilaydi.