mufassall

Konspekt · Ishonchlilik

SLO va xato byudjeti

14 daq · O'rta · mufassall.dtwin.uz/uz/l/slo-error-budget/

← Darsni ochish

Muammo

Taksi ilovasining buyurtma API'si soniyasiga 100 ga yaqin so'rovga xizmat qiladi. Health check 100% ko'rsatadi, yo'lovchilar esa xato ko'radi. Ishonchlilikni foydalanuvchi qanday his qilsa, shunday o'lchash, maqsadni kelishish, farqni xato byudjetiga aylantirish va odamlarni faqat byudjet haqiqatan yonayotganda uyg'otish.

Nimani o'rganasiz

  • SLI, SLO va SLA: nima o'lchanadi, maqsad nima va nima va'da qilinadi
  • Xato byudjeti hisobi: 30 kunda 99,9% — taxminan 43 daqiqa yoki 259 200 ta muvaffaqiyatsiz so'rov
  • Yonish tezligi (burn rate) va ikki oynali alertlar: tez uyg'otadi, sakrashlarda jim turadi
  • Byudjet tugaganda relizlarni muzlatish va persentil bo'yicha kechikish SLO'si
Yakuniy sahna
40%55%10%45%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_riders(["Yo'lovchilar"])
  n_lb{"Balanslovchi"}
  n_api["Buyurtma API"]
  n_db[("Safarlar bazasi")]
  n_monitor["Monitoring"]
  n_pipeline("Reliz pipeline")
  n_pager(("Navbatchi"))
  n_riders --> n_lb
  n_lb --> n_api
  n_api --> n_db
  n_lb --> n_monitor
  n_pipeline --> n_api
  n_monitor --> n_pager
  n_pipeline --> n_monitor

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_monitor as Monitoring
  participant n_api as Buyurtma API
  participant n_lb as Balanslovchi
  participant n_riders as Yo'lovchilar
  participant n_pipeline as Reliz pipeline
  actor n_pager as Navbatchi
  Note over n_monitor,n_api: Health check yashil, yo'lovchilar xato ko'radi
  n_monitor->>n_api: GET /healthz HTTP/1.1
  n_api-->>n_monitor: HTTP/1.1 200 OK
  Note over n_lb,n_monitor: SLI: yaxshi so'rovlarni foydalanuvchi uchragan joyda sanash
  n_lb->>n_monitor: lb access log, POST /rides, 10:00-10:01
  Note over n_lb,n_monitor: SLO: maqsad va u qoldiradigan byudjet
  n_lb->>n_monitor: slo ride-booking
  Note over n_riders,n_lb: SLA: pulga bog'langan va'da
  n_riders->>n_lb: SLA, corporate rides contract
  Note over n_pipeline,n_api: Reliz byudjetni ellik barobar tez yoqadi
  n_pipeline->>n_api: deploy ride-api v412 to 100%
  n_api-->>n_pipeline: error rate 5.0%, the SLO allows 0.1%
  Note over n_monitor,n_pager: Ikki sodda alert: juda shovqinli va juda kech
  n_monitor->>n_pager: rule A: error_rate_5m #62; 0.1% -#62; page
  Note over n_monitor,n_pager: Ikki oynali yonish tezligi alertlari
  n_monitor->>n_pager: ALERT BudgetBurnFast (page)
  n_pager-->>n_monitor: v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
  Note over n_pipeline,n_monitor: Byudjet tugadi: relizlar to'xtaydi
  n_pipeline->>n_monitor: GET /slo/ride-booking/budget HTTP/1.1
  n_monitor-->>n_pipeline: HTTP/1.1 200 OK
  Note over n_lb,n_monitor: Kechikish SLO'si — persentil, o'rtacha emas
  n_lb->>n_monitor: histogram ride_request_duration, last 30 days

D2

direction: right

n_riders: "Yo'lovchilar" {
  shape: rectangle
}
n_lb: "Balanslovchi" {
  shape: diamond
}
n_api: "Buyurtma API" {
  shape: rectangle
}
n_db: "Safarlar bazasi" {
  shape: cylinder
}
n_monitor: "Monitoring" {
  shape: rectangle
}
n_pipeline: "Reliz pipeline" {
  shape: rectangle
  style.multiple: true
}
n_pager: "Navbatchi" {
  shape: person
}

n_riders -> n_lb
n_lb -> n_api
n_api -> n_db
n_lb -> n_monitor
n_pipeline -> n_api
n_monitor -> n_pager
n_pipeline -> n_monitor

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Monitoring" as n_monitor
participant "Buyurtma API" as n_api
participant "Balanslovchi" as n_lb
participant "Yo'lovchilar" as n_riders
participant "Reliz pipeline" as n_pipeline
actor "Navbatchi" as n_pager
note over n_monitor, n_api : Health check yashil, yo'lovchilar xato ko'radi
n_monitor -> n_api : GET /healthz HTTP/1.1
n_api --> n_monitor : HTTP/1.1 200 OK
note over n_lb, n_monitor : SLI: yaxshi so'rovlarni foydalanuvchi uchragan joyda sanash
n_lb -> n_monitor : lb access log, POST /rides, 10:00-10:01
note over n_lb, n_monitor : SLO: maqsad va u qoldiradigan byudjet
n_lb -> n_monitor : slo ride-booking
note over n_riders, n_lb : SLA: pulga bog'langan va'da
n_riders -> n_lb : SLA, corporate rides contract
note over n_pipeline, n_api : Reliz byudjetni ellik barobar tez yoqadi
n_pipeline -> n_api : deploy ride-api v412 to 100%
n_api --> n_pipeline : error rate 5.0%, the SLO allows 0.1%
note over n_monitor, n_pager : Ikki sodda alert: juda shovqinli va juda kech
n_monitor -> n_pager : rule A: error_rate_5m > 0.1% -> page
note over n_monitor, n_pager : Ikki oynali yonish tezligi alertlari
n_monitor -> n_pager : ALERT BudgetBurnFast (page)
n_pager --> n_monitor : v412 replay: paged at +17 min, rolled back at +25 min, 3% of budget spent
note over n_pipeline, n_monitor : Byudjet tugadi: relizlar to'xtaydi
n_pipeline -> n_monitor : GET /slo/ride-booking/budget HTTP/1.1
n_monitor --> n_pipeline : HTTP/1.1 200 OK
note over n_lb, n_monitor : Kechikish SLO'si — persentil, o'rtacha emas
n_lb -> n_monitor : histogram ride_request_duration, last 30 days
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Yo'lovchilar" as n_riders
node "Balanslovchi" as n_lb
component "Buyurtma API" as n_api
database "Safarlar bazasi" as n_db
component "Monitoring" as n_monitor
collections "Reliz pipeline" as n_pipeline
actor "Navbatchi" as n_pager
n_riders --> n_lb
n_lb --> n_api
n_api --> n_db
n_lb --> n_monitor
n_pipeline --> n_api
n_monitor --> n_pager
n_pipeline --> n_monitor
@enduml

Xulosa

SLO uchun olti qoida.

1) SLI'ni foydalanuvchi uchragan joyda o'lchang: yaxshi so'rovlar yaroqlilarga nisbatan, balanslovchida. 2) SLO'ni 100% dan past tanlang; qolgani xato byudjeti (30 kunda 99,9% — taxminan 43 daqiqa). 3) SLA'ni SLO'dan yumshoqroq saqlang. 4) Yonish tezligi bo'yicha uzun va qisqa oyna bilan alert qiling: tez yonishda page, sekinida tiket. 5) Byudjet tugasa feature relizlari muzlashini oldindan kelishing. 6) Kechikish SLO'sini o'rtacha bilan emas, persentil bilan yozing.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Yaxshi so'rovlar98,5%100,0%
Byudjet qoldig'i80%−12%
Yonish tezligi0,5 x0,5 x
Buyurtma p99450 ms900 ms

Qadamlar

  1. Muammo Health check yashil, yo'lovchilar xato ko'radi

    Monitoring har 10 soniyada buyurtma API'sining /healthz manzilini chaqiradi.

  2. Yechim SLI: yaxshi so'rovlarni foydalanuvchi uchragan joyda sanash

    Jamoa raqamni balanslovchi logidan oladi: har bir haqiqiy buyurtma so'rovi va u yaxshi bo'lganmi.

  3. Yechim SLO: maqsad va u qoldiradigan byudjet

    Jamoa SLO ni kelishadi: siljuvchi 30 kun ichida buyurtma so'rovlarining 99,9% i yaxshi.

  4. Holat SLA: pulga bog'langan va'da

    Korporativ mijozlar SLA, ya'ni shartnoma oladi: oyiga 99,5%, aks holda taksi kompaniyasi hisobning bir qismini qaytaradi.

  5. Muammo Reliz byudjetni ellik barobar tez yoqadi

    v412 reliz tarif keshini qo'shadi.

  6. Muammo Ikki sodda alert: juda shovqinli va juda kech

    A qoida xato ulushi 5 daqiqa davomida 0,1% dan oshsa uyg'otadi.

  7. Yechim Ikki oynali yonish tezligi alertlari

    Endi page yonish tezligi oxirgi bir soatda ham, oxirgi 5 daqiqada ham 14,4 dan yuqori bo'lsa ishlaydi.

  8. Yechim Byudjet tugadi: relizlar to'xtaydi

    Oyning keyingi qismida baza migratsiyasi va provayder uzilishi byudjetdan oshib ketadi: 12% ortiqcha.

  9. Holat Kechikish SLO'si — persentil, o'rtacha emas

    Muvaffaqiyat faqat "xato yo'q" degani emas.

  10. Xulosa Xulosa

    SLO uchun olti qoida.

Atamalar

SLI (xizmat darajasi ko'rsatkichi)
Foydalanuvchi sezadigan narsaning o'lchovi: yaxshi so'rovlar ulushi yoki ma'lum vaqtdan tez tugaganlar ulushi. Masalan, "300 ms dan tez va xatosiz javoblar foizi".
SLO (xizmat darajasi maqsadi)
SLI uchun ichki maqsad va davr: "28 kunda so'rovlarning 99,9% i yaxshi". 100% maqsad qilinmaydi: har bir qo'shimcha to'qqiz narxi keskin oshadi, foydalanuvchi esa farqni sezmaydi.
SLA (xizmat darajasi shartnomasi)
Mijoz bilan shartnomadagi va'da: buzilsa pul qaytariladi yoki chegirma beriladi. Odatda ichki SLO'dan yumshoqroq qo'yiladi, toki jamoa shartnoma buzilishidan oldin xabar topsin.
Xato byudjeti (error budget)
SLO ruxsat bergan yomon so'rovlar ulushi: 99,9% da 0,1%, ya'ni 30 kunda taxminan 43 daqiqa to'liq uzilish. Byudjet qolsa reliz chiqariladi, tugasa relizlar to'xtatilib ishonchlilikka ishlanadi.
Yonish tezligi (burn rate)
Xato byudjeti me'yordan necha barobar tez sarflanayotgani: 1 — davr oxirida aynan tugaydi, 14,4 — 30 kunlik byudjetning 2% i bir soatda ketadi. Alert ikki oynada (uzun va qisqa) birga tekshiriladi.