Konspekt · Butun tizimlar
Job scheduler (vazifa rejalashtiruvchi) dizayni
Muammo
"Dayjestni 09:00 da yubor", "ertaga eslat", "har 5 daqiqada sinxronla": ko'p tenant uchun millionlab kechiktirilgan va takrorlanuvchi vazifalar. Bitta cron serverdan `SKIP LOCKED` bilan olinadigan jobs jadvaliga, heartbeat'li lease'larga, idempotent at-least-once ishga tushirishlarga, backoff'li qayta urinishlar va dead-letter jadvaliga o'tamiz, keyin 09:00 cho'qqisini vaqt partitsiyalari va tenant'lar o'rtasidagi adolat bilan yengamiz.
Nimani o'rganasiz
- Nega bitta cron server ham vazifalarni o'tkazib yuboradi, ham failover'dan keyin ikki marta ishga tushiradi
- Worker'lar vaqti kelgan qatorlarni
FOR UPDATE SKIP LOCKEDbilan qanday xavfsiz oladi - Lease va heartbeat: nega ular at-least-once beradi va nega buning uchun idempotency kaliti kerak
- Jitter'li backoff, dead-letter jadvali, vaqt partitsiyalari va tenant'lar o'rtasida adolat
Diagramma kod sifatida
Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.
Mermaid: sxema
flowchart LR
n_app(["Tenant ilovasi"])
n_mail>"Email provayder"]
n_api["Scheduler API"]
n_db[("Jobs jadvali")]
n_w1("Worker 1")
n_w2("Worker 2")
n_dlq[("O'lik job'lar")]
n_app2(["Katta tenant"])
n_app --> n_api
n_api --> n_db
n_w1 --> n_db
n_w2 --> n_db
n_w1 --> n_mail
n_w2 --> n_mail
n_w2 --x n_dlq
n_app2 --> n_apiMermaid: so'rovlar ketma-ketligi
sequenceDiagram
participant n_app as Tenant ilovasi
participant n_cron as Cron server A
participant n_cron2 as Cron server B
participant n_mail as Email provayder
participant n_api as Scheduler API
participant n_db as Jobs jadvali
participant n_w1 as Worker 1
participant n_w2 as Worker 2
participant n_dlq as O'lik job'lar
Note over n_app,n_cron: Bitta mashina, bitta crontab
n_app->>n_cron: $ crontab -l #35; on cron-a
Note over n_cron2,n_mail: Server o'ladi, keyin ikki marta qaytadi
n_cron2->>n_mail: 08:12 cron-a unreachable, jobs 08:15..08:50 never ran
Note over n_api,n_db: Jadval vaqti yozilgan qatorlarga aylanadi
n_api->>n_db: INSERT INTO jobs -- the row, as JSON
n_db-->>n_api: INSERT 0 1
Note over n_w1,n_db: Qatorlarni FOR UPDATE SKIP LOCKED bilan olish
n_w1->>n_db: WITH picked AS (
n_db-->>n_w1: 50 rows, 3.1 ms
Note over n_w1,n_db: O'lgan worker o'z job'larini osilib qoldiradi
n_w1->>n_db: SELECT id, locked_by, now() - updated_at AS stuck
n_db-->>n_w1: j_8840 #124; w-2 #124; 00:47:12
Note over n_w1,n_db: Yangilanmasa muddati tugaydigan lease
n_w1->>n_db: UPDATE jobs
n_db-->>n_w1: lease_until = 09:00:41+05
Note over n_w1,n_mail: Har ishga tushirish idempotency kaliti bilan
n_w1->>n_mail: POST /v1/messages HTTP/1.1
n_mail-->>n_w1: HTTP/1.1 200 OK
Note over n_w2,n_mail: Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi
n_w2->>n_mail: POST /v1/messages HTTP/1.1
n_mail-->>n_w2: HTTP/1.1 503 Service Unavailable
Note over n_w2,n_dlq: Eksponensial backoff, jitter va dead-letter jadvali
n_w2->>n_dlq: 09:41:07 WARN job j_9310 attempt 8/8: 550 mailbox does not exist
Note over n_w2,n_db: Hamma aynan 09:00 ni xohlaydi
n_w2->>n_db: SELECT tenant, count(*) AS due, max(now() - run_at) AS late
n_db-->>n_w2: t_9 1 480 220 00:14:51
Note over n_w1,n_db: Soatlik partitsiyalar va har tenant'ga ulush
n_w1->>n_db: SELECT j.id FROM active_tenants t
n_db-->>n_w1: 50 rows from 10 tenants, 2.4 msD2
direction: right
n_app: "Tenant ilovasi" {
shape: rectangle
}
n_mail: "Email provayder" {
shape: cloud
}
n_api: "Scheduler API" {
shape: rectangle
}
n_db: "Jobs jadvali" {
shape: cylinder
}
n_w1: "Worker 1" {
shape: rectangle
style.multiple: true
}
n_w2: "Worker 2" {
shape: rectangle
style.multiple: true
}
n_dlq: "O'lik job'lar" {
shape: cylinder
}
n_app2: "Katta tenant" {
shape: rectangle
}
n_app -> n_api
n_api -> n_db
n_w1 -> n_db
n_w2 -> n_db
n_w1 -> n_mail
n_w2 -> n_mail
n_w2 -> n_dlq {
target-arrowhead.shape: cross
}
n_app2 -> n_apiPlantUML: so'rovlar ketma-ketligi
@startuml
participant "Tenant ilovasi" as n_app
participant "Cron server A" as n_cron
participant "Cron server B" as n_cron2
boundary "Email provayder" as n_mail
participant "Scheduler API" as n_api
database "Jobs jadvali" as n_db
participant "Worker 1" as n_w1
participant "Worker 2" as n_w2
database "O'lik job'lar" as n_dlq
note over n_app, n_cron : Bitta mashina, bitta crontab
n_app -> n_cron : $ crontab -l # on cron-a
note over n_cron2, n_mail : Server o'ladi, keyin ikki marta qaytadi
n_cron2 -> n_mail : [dublikat] 08:12 cron-a unreachable, jobs 08:15..08:50 never ran
note over n_api, n_db : Jadval vaqti yozilgan qatorlarga aylanadi
n_api -> n_db : INSERT INTO jobs -- the row, as JSON
n_db --> n_api : INSERT 0 1
note over n_w1, n_db : Qatorlarni FOR UPDATE SKIP LOCKED bilan olish
n_w1 -> n_db : WITH picked AS (
n_db --> n_w1 : 50 rows, 3.1 ms
note over n_w1, n_db : O'lgan worker o'z job'larini osilib qoldiradi
n_w1 -> n_db : SELECT id, locked_by, now() - updated_at AS stuck
n_db --> n_w1 : j_8840 | w-2 | 00:47:12
note over n_w1, n_db : Yangilanmasa muddati tugaydigan lease
n_w1 -> n_db : UPDATE jobs
n_db --> n_w1 : lease_until = 09:00:41+05
note over n_w1, n_mail : Har ishga tushirish idempotency kaliti bilan
n_w1 -> n_mail : POST /v1/messages HTTP/1.1
n_mail --> n_w1 : HTTP/1.1 200 OK
note over n_w2, n_mail : Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi
n_w2 -> n_mail : [qayta urinish] POST /v1/messages HTTP/1.1
n_mail --> n_w2 : HTTP/1.1 503 Service Unavailable
note over n_w2, n_dlq : Eksponensial backoff, jitter va dead-letter jadvali
n_w2 -> n_dlq : 09:41:07 WARN job j_9310 attempt 8/8: 550 mailbox does not exist
note over n_w2, n_db : Hamma aynan 09:00 ni xohlaydi
n_w2 -> n_db : SELECT tenant, count(*) AS due, max(now() - run_at) AS late
n_db --> n_w2 : t_9 1 480 220 00:14:51
note over n_w1, n_db : Soatlik partitsiyalar va har tenant'ga ulush
n_w1 -> n_db : SELECT j.id FROM active_tenants t
n_db --> n_w1 : 50 rows from 10 tenants, 2.4 ms
@endumlPlantUML: komponentlar
@startuml
left to right direction
agent "Tenant ilovasi" as n_app
cloud "Email provayder" as n_mail
component "Scheduler API" as n_api
database "Jobs jadvali" as n_db
collections "Worker 1" as n_w1
collections "Worker 2" as n_w2
database "O'lik job'lar" as n_dlq
agent "Katta tenant" as n_app2
n_app --> n_api
n_api --> n_db
n_w1 --> n_db
n_w2 --> n_db
n_w1 --> n_mail
n_w2 --> n_mail
n_w2 -[#c62828]-> n_dlq
n_app2 --> n_api
@endumlXulosa
Scheduler bu vaqtlar bazasi va tirik qolishiga ishonib bo'lmaydigan worker'lar.
1) Bitta cron server o'chiqligida job'larni o'tkazib yuboradi, failover'dan keyin ikki marta bajaradi. 2) Job'larni run_at UTC da yozilgan qatorlar sifatida saqlang; worker'lar FOR UPDATE SKIP LOCKED bilan oladi. 3) Olish heartbeat bilan yangilanadigan lease, baza soati bo'yicha; muddat tugashi osilib qolgan job'larni qaytaradi. 4) Bu ishga tushirishni at least once qiladi, shuning uchun har tashqi ta'sir idempotency kaliti bilan boradi. 5) Eksponensial backoff va jitter bilan qayta rejalashtirib qayta urining; qolganini dead-letter jadvaliga qo'ying va ogohlantirish qo'ying. 6) Vaqt bo'yicha partitsiyalang, eski partitsiyalarni o'chiring va 09:00 adolatli bo'lishi uchun har tenant'dan ulush oling.
Ko'rsatkichlar: boshida va oxirida
| Ko'rsatkich | Boshida | Oxirida |
|---|---|---|
| Boshlanish kechikishi p99 | 1 s | 3 s |
| Bajarilmagan job | 0/kun | 0/kun |
| Ikki marta bajarilgan | 0/kun | 0/kun |
Qadamlar
- Holat Bitta mashina, bitta crontab
Tenant'lar ilova orqali ish rejalashtiradi: har kuni 09:00 da dayjest, har besh daqiqada kurs sinxroni, ertaga bir martalik eslatma.
- Muammo Server o'ladi, keyin ikki marta qaytadi
08:12 da cron-a tarmoqni yo'qotadi.
- Yechim Jadval vaqti yozilgan qatorlarga aylanadi
Endi API har job'ni qator qilib qo'shadi: nima bajariladi, qaysi tenant uchun va
run_at, UTC da. - Yechim Qatorlarni
FOR UPDATE SKIP LOCKEDbilan olishHar worker bitta so'rovda bir to'plamni oladi.
- Muammo O'lgan worker o'z job'larini osilib qoldiradi
Worker 2 09:00:05 da 50 ta job olgan holda o'ldiriladi.
- Yechim Yangilanmasa muddati tugaydigan lease
Endi olish lease beradi:
lease_until30 soniya oldinda. - Yechim Har ishga tushirish idempotency kaliti bilan
Worker har emailni job id va rejalashtirilgan vaqtdan tuzilgan kalit bilan yuboradi:
j_8812:2026-11-19T04:00Z. - Muammo Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi
09:00 da email provayder o'n daqiqa davomida
503qaytara boshlaydi. - Yechim Eksponensial backoff, jitter va dead-letter jadvali
Muvaffaqiyatsiz job
run_atsurilgan holdaqueuedga qaytadi: 30 s, 1 daqiqa, 2 daqiqa, 32 daqiqagacha ikki baravardan, har kechikish 20% gacha tasodifiy siljiydi. - Muammo Hamma aynan 09:00 ni xohlaydi
Ko'pchilik yaxlit vaqtni tanlaydi, bitta katta tenant
t_9esa 09:00 ga 1.5 million eslatma rejalashtirgan. - Yechim Soatlik partitsiyalar va har tenant'ga ulush
Jobs jadvali
run_atsoati bo'yicha partitsiyalangan. - Xulosa Xulosa
Scheduler bu vaqtlar bazasi va tirik qolishiga ishonib bo'lmaydigan worker'lar.
Atamalar
- SKIP LOCKED
- FOR UPDATE ga qo'shimcha: boshqa tranzaksiya qulflagan qatorlarni kutmasdan o'tkazib yuboradi, worker'lar bir-birini to'smaydi.
- Lease (ijara muddati)
- Ishga vaqtincha egalik huquqi; egasi heartbeat bilan yangilab turmasa, muddati tugaydi va ish boshqasiga o'tadi.
- Vaqt bo'yicha partitsiya
- Jadvalni soat yoki kun bo'yicha bo'laklarga ajratish; so'rovlar faqat joriy bo'lakka tegadi, eskisi butunlay o'chiriladi.