Konspekt · Ishonchlilik
Kuzatuvchanlik (observability)
Muammo
Buyurtma 2,4 soniya oladi, har servis esa o'zida hammasi joyida deydi. Log, servislararo request id, RED va USE metrikalar, p95 bo'yicha alert va taqsimlangan trace sekin joyni qanday topadi.
Nimani o'rganasiz
- Log, metrika va trace har biri qaysi savolga javob beradi, qaysiga yo'q
- Nega bitta request id yoki trace id hamma servisdan o'tishi kerak
- Servislar uchun RED, resurslar uchun USE va p95 bo'yicha alert
- Trace'da sekin span'ni topish va sampling bilan narxni pasaytirish
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_client(["Mijoz ilovasi"])
n_api["API"]
n_orders["Buyurtmalar servisi"]
n_stock["Qoldiqlar servisi"]
n_db[("Buyurtmalar bazasi")]
n_logs[("Log ombori")]
n_tsdb[("Metrikalar bazasi")]
n_oncall(("Navbatchi muhandis"))
n_tracer[("Trace ombori")]
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb -.-> n_oncall
n_api --> n_tracer
n_orders --> n_tracerMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_client as Mijoz ilovasi
participant n_api as API
participant n_orders as Buyurtmalar servisi
participant n_logs as Log ombori
participant n_tsdb as Metrikalar bazasi
participant n_db as Buyurtmalar bazasi
actor n_oncall as Navbatchi muhandis
participant n_tracer as Trace ombori
participant n_stock as Qoldiqlar servisi
Note over n_client,n_api: Sekin, lekin hech narsa buzilmagan
n_client->>n_api: POST /v1/orders HTTP/1.1
n_api-->>n_client: HTTP/1.1 201 Created
Note over n_orders,n_logs: Strukturali log bitta joyda
n_orders->>n_logs: orders stdout → log shipper
Note over n_api,n_orders: Butun so'rov uchun bitta id
n_api->>n_orders: POST /internal/orders HTTP/1.1
n_orders-->>n_api: log line in orders
Note over n_orders,n_tsdb: RED metrikalar: rate, errors, duration
n_orders->>n_tsdb: GET /metrics HTTP/1.1
n_tsdb-->>n_orders: HTTP/1.1 200 OK
Note over n_db,n_tsdb: USE metrikalar: baza aybdormi?
n_db->>n_tsdb: GET /metrics HTTP/1.1
n_tsdb-->>n_db: HTTP/1.1 200 OK
Note over n_tsdb,n_oncall: O'rtacha emas, p95 bo'yicha alert
n_tsdb->>n_oncall: ALERT OrdersSlowP95 firing
Note over n_api,n_orders: Trace: har qadamdagi vaqt
n_api->>n_orders: POST /internal/orders HTTP/1.1
Note over n_orders,n_tracer: Sekin span'ni topamiz
n_orders->>n_tracer: trace 4bf92f35 POST /v1/orders 2310ms
Note over n_orders,n_stock: Butun savat uchun bitta chaqiruv
n_orders->>n_stock: GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock-->>n_orders: HTTP/1.1 200 OK
Note over n_api,n_tracer: Trace qimmat: sampling qilamiz
n_api->>n_tracer: sampler: keep 10% of traces, all errors, all over 1sD2
direction: right
n_client: "Mijoz ilovasi" {
shape: rectangle
}
n_api: "API" {
shape: rectangle
}
n_orders: "Buyurtmalar servisi" {
shape: rectangle
}
n_stock: "Qoldiqlar servisi" {
shape: rectangle
}
n_db: "Buyurtmalar bazasi" {
shape: cylinder
}
n_logs: "Log ombori" {
shape: stored_data
}
n_tsdb: "Metrikalar bazasi" {
shape: cylinder
}
n_oncall: "Navbatchi muhandis" {
shape: person
}
n_tracer: "Trace ombori" {
shape: stored_data
}
n_client -> n_api
n_api -> n_orders
n_orders -> n_stock
n_orders -> n_db
n_api -> n_logs
n_orders -> n_logs
n_api -> n_tsdb
n_orders -> n_tsdb
n_db -> n_tsdb
n_tsdb -> n_oncall {
style.stroke-dash: 3
}
n_api -> n_tracer
n_orders -> n_tracerPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Mijoz ilovasi" as n_client
participant "API" as n_api
participant "Buyurtmalar servisi" as n_orders
database "Log ombori" as n_logs
database "Metrikalar bazasi" as n_tsdb
database "Buyurtmalar bazasi" as n_db
actor "Navbatchi muhandis" as n_oncall
database "Trace ombori" as n_tracer
participant "Qoldiqlar servisi" as n_stock
note over n_client, n_api : Sekin, lekin hech narsa buzilmagan
n_client -> n_api : POST /v1/orders HTTP/1.1
n_api --> n_client : HTTP/1.1 201 Created
note over n_orders, n_logs : Strukturali log bitta joyda
n_orders -> n_logs : orders stdout → log shipper
note over n_api, n_orders : Butun so'rov uchun bitta id
n_api -> n_orders : POST /internal/orders HTTP/1.1
n_orders --> n_api : log line in orders
note over n_orders, n_tsdb : RED metrikalar: rate, errors, duration
n_orders -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_orders : HTTP/1.1 200 OK
note over n_db, n_tsdb : USE metrikalar: baza aybdormi?
n_db -> n_tsdb : GET /metrics HTTP/1.1
n_tsdb --> n_db : HTTP/1.1 200 OK
note over n_tsdb, n_oncall : O'rtacha emas, p95 bo'yicha alert
n_tsdb -> n_oncall : ALERT OrdersSlowP95 firing
note over n_api, n_orders : Trace: har qadamdagi vaqt
n_api -> n_orders : POST /internal/orders HTTP/1.1
note over n_orders, n_tracer : Sekin span'ni topamiz
n_orders -> n_tracer : trace 4bf92f35 POST /v1/orders 2310ms
note over n_orders, n_stock : Butun savat uchun bitta chaqiruv
n_orders -> n_stock : GET /v1/stock?skus=sku_19,sku_20,...,sku_30 HTTP/1.1
n_stock --> n_orders : HTTP/1.1 200 OK
note over n_api, n_tracer : Trace qimmat: sampling qilamiz
n_api -> n_tracer : sampler: keep 10% of traces, all errors, all over 1s
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Mijoz ilovasi" as n_client
component "API" as n_api
component "Buyurtmalar servisi" as n_orders
component "Qoldiqlar servisi" as n_stock
database "Buyurtmalar bazasi" as n_db
storage "Log ombori" as n_logs
database "Metrikalar bazasi" as n_tsdb
actor "Navbatchi muhandis" as n_oncall
storage "Trace ombori" as n_tracer
n_client --> n_api
n_api --> n_orders
n_orders --> n_stock
n_orders --> n_db
n_api --> n_logs
n_orders --> n_logs
n_api --> n_tsdb
n_orders --> n_tsdb
n_db --> n_tsdb
n_tsdb ..> n_oncall
n_api --> n_tracer
n_orders --> n_tracer
@endumlXulosa
Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi.
1) Strukturali JSON'ni bitta joyga yozing. 2) Kirishda bitta request id yoki trace id yarating va uni hamma joyga uzating. 3) Servislarga RED, resurslarga USE. 4) O'rtachaga emas, bir necha daqiqalik p95 va xatolarga alert. 5) Sekin span'ni trace'da toping va narxni ushlash uchun sampling qiling.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Buyurtma p95 | 2 400 ms | 420 ms |
| Sababni topish vaqti | 5 400 s | 120 s |
| Kunlik telemetriya | 0,4 GB | 2,5 GB |
Qadamlar
- Muammo Sekin, lekin hech narsa buzilmagan
Mijozlar buyurtma «osilib qoladi» deyishadi.
- Yechim Strukturali log bitta joyda
Endi har servis har hodisa uchun stdout'ga bitta JSON obyekt yozadi.
- Yechim Butun so'rov uchun bitta id
So'rov kelganda API request id yaratadi va uni header'da keyingi servisga uzatadi.
- Yechim RED metrikalar: rate, errors, duration
Log alohida hodisalarni tasvirlaydi.
- Holat USE metrikalar: baza aybdormi?
Resurslar uchun (baza, disk, pool) USE'ga qaraladi: Utilisation, Saturation, Errors.
- Yechim O'rtacha emas, p95 bo'yicha alert
Hozirgacha birinchi bo'lib mijozlar sezardi.
- Yechim Trace: har qadamdagi vaqt
Request id trace'ga aylanadi.
- Holat Sekin span'ni topamiz
Bitta sekin trace'ni ochamiz.
- Yechim Butun savat uchun bitta chaqiruv
Endi orders qoldiqlarni savatdagi barcha tovarlar uchun bir marta so'raydi.
- Holat Trace qimmat: sampling qilamiz
Har so'rovning trace'i kuniga 12 GB beradi, asosan tez va zerikarli so'rovlar.
- Xulosa Xulosa
Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi.
Atamalar
- Observability (log, metrika, trace)
- Log nima bo'lganini, metrika qancha va qanchalik tez-tez bo'lganini, trace vaqt qayerga ketganini aytadi. Servislar uchun RED, resurslar uchun USE metrikalari, alert esa o'rtachaga emas, p95 ga.
- Trace va span
- Har servis bitta so'rov uchun span yozadi: nima qilgan, qachon boshlagan va qancha davom etgan. Span'lar umumiy trace id orqali bitta daraxtga yig'iladi va sekin qadam ko'rinadi.