mufassall

Konspekt · Ishonchlilik

Kuzatuvchanlik (observability)

13 daq · O'rta · mufassall.dtwin.uz/uz/l/observability/

← Darsni ochish

Muammo

Buyurtma 2,4 soniya oladi, har servis esa o'zida hammasi joyida deydi. Log, servislararo request id, RED va USE metrikalar, p95 bo'yicha alert va taqsimlangan trace sekin joyni qanday topadi.

Nimani o'rganasiz

  • Log, metrika va trace har biri qaysi savolga javob beradi, qaysiga yo'q
  • Nega bitta request id yoki trace id hamma servisdan o'tishi kerak
  • Servislar uchun RED, resurslar uchun USE va p95 bo'yicha alert
  • Trace'da sekin span'ni topish va sampling bilan narxni pasaytirish
Yakuniy sahna
35%30%41%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_client(["Mijoz ilovasi"])
  n_api["API"]
  n_orders["Buyurtmalar servisi"]
  n_stock["Qoldiqlar servisi"]
  n_db[("Buyurtmalar bazasi")]
  n_logs[("Log ombori")]
  n_tsdb[("Metrikalar bazasi")]
  n_oncall(("Navbatchi muhandis"))
  n_tracer[("Trace ombori")]
  n_client --> n_api
  n_api --> n_orders
  n_orders --> n_stock
  n_orders --> n_db
  n_api --> n_logs
  n_orders --> n_logs
  n_api --> n_tsdb
  n_orders --> n_tsdb
  n_db --> n_tsdb
  n_tsdb -.-> n_oncall
  n_api --> n_tracer
  n_orders --> n_tracer

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_client as Mijoz ilovasi
  participant n_api as API
  participant n_orders as Buyurtmalar servisi
  participant n_logs as Log ombori
  participant n_tsdb as Metrikalar bazasi
  participant n_db as Buyurtmalar bazasi
  actor n_oncall as Navbatchi muhandis
  participant n_tracer as Trace ombori
  participant n_stock as Qoldiqlar servisi
  Note over n_client,n_api: Sekin, lekin hech narsa buzilmagan
  n_client->>n_api: POST /v1/orders HTTP/1.1
  n_api-->>n_client: HTTP/1.1 201 Created
  Note over n_orders,n_logs: Strukturali log bitta joyda
  n_orders->>n_logs: orders stdout → log shipper
  Note over n_api,n_orders: Butun so'rov uchun bitta id
  n_api->>n_orders: POST /internal/orders HTTP/1.1
  n_orders-->>n_api: log line in orders
  Note over n_orders,n_tsdb: RED metrikalar: rate, errors, duration
  n_orders->>n_tsdb: GET /metrics HTTP/1.1
  n_tsdb-->>n_orders: HTTP/1.1 200 OK
  Note over n_db,n_tsdb: USE metrikalar: baza aybdormi?
  n_db->>n_tsdb: GET /metrics HTTP/1.1
  n_tsdb-->>n_db: HTTP/1.1 200 OK
  Note over n_tsdb,n_oncall: O'rtacha emas, p95 bo'yicha alert
  n_tsdb->>n_oncall: ALERT OrdersSlowP95 firing
  Note over n_api,n_orders: Trace: har qadamdagi vaqt
  n_api->>n_orders: POST /internal/orders HTTP/1.1
  Note over n_orders,n_tracer: Sekin span'ni topamiz
  n_orders->>n_tracer: trace 4bf92f35 POST /v1/orders 2310ms
  Note over n_orders,n_stock: Butun savat uchun bitta chaqiruv
  n_orders->>n_stock: GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
  n_stock-->>n_orders: HTTP/1.1 200 OK
  Note over n_api,n_tracer: Trace qimmat: sampling qilamiz
  n_api->>n_tracer: sampler: keep 10% of traces, all errors, all over 1s

D2

direction: right

n_client: "Mijoz ilovasi" {
  shape: rectangle
}
n_api: "API" {
  shape: rectangle
}
n_orders: "Buyurtmalar servisi" {
  shape: rectangle
}
n_stock: "Qoldiqlar servisi" {
  shape: rectangle
}
n_db: "Buyurtmalar bazasi" {
  shape: cylinder
}
n_logs: "Log ombori" {
  shape: stored_data
}
n_tsdb: "Metrikalar bazasi" {
  shape: cylinder
}
n_oncall: "Navbatchi muhandis" {
  shape: person
}
n_tracer: "Trace ombori" {
  shape: stored_data
}

n_client -> n_api
n_api -> n_orders
n_orders -> n_stock
n_orders -> n_db
n_api -> n_logs
n_orders -> n_logs
n_api -> n_tsdb
n_orders -> n_tsdb
n_db -> n_tsdb
n_tsdb -> n_oncall {
  style.stroke-dash: 3
}
n_api -> n_tracer
n_orders -> n_tracer

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Mijoz ilovasi" as n_client
participant "API" as n_api
participant "Buyurtmalar servisi" as n_orders
database "Log ombori" as n_logs
database "Metrikalar bazasi" as n_tsdb
database "Buyurtmalar bazasi" as n_db
actor "Navbatchi muhandis" as n_oncall
database "Trace ombori" as n_tracer
participant "Qoldiqlar servisi" as n_stock
note over n_client, n_api : Sekin, lekin hech narsa buzilmagan
n_client -> n_api : POST /v1/orders HTTP/1.1
n_api --> n_client : HTTP/1.1 201 Created
note over n_orders, n_logs : Strukturali log bitta joyda
n_orders -> n_logs : orders stdout → log shipper
note over n_api, n_orders : Butun so'rov uchun bitta id
n_api -> n_orders : POST /internal/orders HTTP/1.1
n_orders --> n_api : log line in orders
note over n_orders, n_tsdb : RED metrikalar: rate, errors, duration
n_orders -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_orders : HTTP/1.1 200 OK
note over n_db, n_tsdb : USE metrikalar: baza aybdormi?
n_db -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_db : HTTP/1.1 200 OK
note over n_tsdb, n_oncall : O'rtacha emas, p95 bo'yicha alert
n_tsdb -> n_oncall : ALERT OrdersSlowP95 firing
note over n_api, n_orders : Trace: har qadamdagi vaqt
n_api -> n_orders : POST /internal/orders HTTP/1.1
note over n_orders, n_tracer : Sekin span'ni topamiz
n_orders -> n_tracer : trace 4bf92f35 POST /v1/orders 2310ms
note over n_orders, n_stock : Butun savat uchun bitta chaqiruv
n_orders -> n_stock : GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock --> n_orders : HTTP/1.1 200 OK
note over n_api, n_tracer : Trace qimmat: sampling qilamiz
n_api -> n_tracer : sampler: keep 10% of traces, all errors, all over 1s
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Mijoz ilovasi" as n_client
component "API" as n_api
component "Buyurtmalar servisi" as n_orders
component "Qoldiqlar servisi" as n_stock
database "Buyurtmalar bazasi" as n_db
storage "Log ombori" as n_logs
database "Metrikalar bazasi" as n_tsdb
actor "Navbatchi muhandis" as n_oncall
storage "Trace ombori" as n_tracer
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb ..> n_oncall
n_api --> n_tracer
n_orders --> n_tracer
@enduml

Xulosa

Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi.

1) Strukturali JSON'ni bitta joyga yozing. 2) Kirishda bitta request id yoki trace id yarating va uni hamma joyga uzating. 3) Servislarga RED, resurslarga USE. 4) O'rtachaga emas, bir necha daqiqalik p95 va xatolarga alert. 5) Sekin span'ni trace'da toping va narxni ushlash uchun sampling qiling.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Buyurtma p952 400 ms420 ms
Sababni topish vaqti5 400 s120 s
Kunlik telemetriya0,4 GB2,5 GB

Qadamlar

  1. Muammo Sekin, lekin hech narsa buzilmagan

    Mijozlar buyurtma «osilib qoladi» deyishadi.

  2. Yechim Strukturali log bitta joyda

    Endi har servis har hodisa uchun stdout'ga bitta JSON obyekt yozadi.

  3. Yechim Butun so'rov uchun bitta id

    So'rov kelganda API request id yaratadi va uni header'da keyingi servisga uzatadi.

  4. Yechim RED metrikalar: rate, errors, duration

    Log alohida hodisalarni tasvirlaydi.

  5. Holat USE metrikalar: baza aybdormi?

    Resurslar uchun (baza, disk, pool) USE'ga qaraladi: Utilisation, Saturation, Errors.

  6. Yechim O'rtacha emas, p95 bo'yicha alert

    Hozirgacha birinchi bo'lib mijozlar sezardi.

  7. Yechim Trace: har qadamdagi vaqt

    Request id trace'ga aylanadi.

  8. Holat Sekin span'ni topamiz

    Bitta sekin trace'ni ochamiz.

  9. Yechim Butun savat uchun bitta chaqiruv

    Endi orders qoldiqlarni savatdagi barcha tovarlar uchun bir marta so'raydi.

  10. Holat Trace qimmat: sampling qilamiz

    Har so'rovning trace'i kuniga 12 GB beradi, asosan tez va zerikarli so'rovlar.

  11. Xulosa Xulosa

    Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi.

Atamalar

Observability (log, metrika, trace)
Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi. Servislar uchun RED, resurslar uchun USE metrikalari, alert esa o'rtachaga emas, p95 ga.
Trace va span
Har servis bitta so'rov uchun span yozadi: nima qilgan, qachon boshlagan va qancha davom etgan. Span'lar umumiy trace id orqali bitta daraxtga yig'iladi va sekin qadam ko'rinadi.