mufassall

Konspekt · Butun tizimlar

Job scheduler (vazifa rejalashtiruvchi) dizayni

15 daq · Yuqori · mufassall.dtwin.uz/uz/l/design-job-scheduler/

← Darsni ochish

Muammo

"Dayjestni 09:00 da yubor", "ertaga eslat", "har 5 daqiqada sinxronla": ko'p tenant uchun millionlab kechiktirilgan va takrorlanuvchi vazifalar. Bitta cron serverdan `SKIP LOCKED` bilan olinadigan jobs jadvaliga, heartbeat'li lease'larga, idempotent at-least-once ishga tushirishlarga, backoff'li qayta urinishlar va dead-letter jadvaliga o'tamiz, keyin 09:00 cho'qqisini vaqt partitsiyalari va tenant'lar o'rtasidagi adolat bilan yengamiz.

Nimani o'rganasiz

  • Nega bitta cron server ham vazifalarni o'tkazib yuboradi, ham failover'dan keyin ikki marta ishga tushiradi
  • Worker'lar vaqti kelgan qatorlarni FOR UPDATE SKIP LOCKED bilan qanday xavfsiz oladi
  • Lease va heartbeat: nega ular at-least-once beradi va nega buning uchun idempotency kaliti kerak
  • Jitter'li backoff, dead-letter jadvali, vaqt partitsiyalari va tenant'lar o'rtasida adolat
Yakuniy sahna
45%50%45%45%

Diagramma kod sifatida

Darsdagi yakuniy sxemani matn ko'rinishida oling va Jira, GitLab, GitHub, Notion yoki hujjatga qo'ying: Mermaid u yerlarda o'zi chiziladi. D2 va PlantUML (Confluence, IntelliJ) uchun alohida variantlar bor.

Mermaid: sxema

flowchart LR
  n_app(["Tenant ilovasi"])
  n_mail>"Email provayder"]
  n_api["Scheduler API"]
  n_db[("Jobs jadvali")]
  n_w1("Worker 1")
  n_w2("Worker 2")
  n_dlq[("O'lik job'lar")]
  n_app2(["Katta tenant"])
  n_app --> n_api
  n_api --> n_db
  n_w1 --> n_db
  n_w2 --> n_db
  n_w1 --> n_mail
  n_w2 --> n_mail
  n_w2 --x n_dlq
  n_app2 --> n_api

Mermaid: so'rovlar ketma-ketligi

sequenceDiagram
  participant n_app as Tenant ilovasi
  participant n_cron as Cron server A
  participant n_cron2 as Cron server B
  participant n_mail as Email provayder
  participant n_api as Scheduler API
  participant n_db as Jobs jadvali
  participant n_w1 as Worker 1
  participant n_w2 as Worker 2
  participant n_dlq as O'lik job'lar
  Note over n_app,n_cron: Bitta mashina, bitta crontab
  n_app->>n_cron: $ crontab -l #35; on cron-a
  Note over n_cron2,n_mail: Server o'ladi, keyin ikki marta qaytadi
  n_cron2->>n_mail: 08:12 cron-a unreachable, jobs 08:15..08:50 never ran
  Note over n_api,n_db: Jadval vaqti yozilgan qatorlarga aylanadi
  n_api->>n_db: INSERT INTO jobs -- the row, as JSON
  n_db-->>n_api: INSERT 0 1
  Note over n_w1,n_db: Qatorlarni FOR UPDATE SKIP LOCKED bilan olish
  n_w1->>n_db: WITH picked AS (
  n_db-->>n_w1: 50 rows, 3.1 ms
  Note over n_w1,n_db: O'lgan worker o'z job'larini osilib qoldiradi
  n_w1->>n_db: SELECT id, locked_by, now() - updated_at AS stuck
  n_db-->>n_w1: j_8840 #124; w-2 #124; 00:47:12
  Note over n_w1,n_db: Yangilanmasa muddati tugaydigan lease
  n_w1->>n_db: UPDATE jobs
  n_db-->>n_w1: lease_until = 09:00:41+05
  Note over n_w1,n_mail: Har ishga tushirish idempotency kaliti bilan
  n_w1->>n_mail: POST /v1/messages HTTP/1.1
  n_mail-->>n_w1: HTTP/1.1 200 OK
  Note over n_w2,n_mail: Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi
  n_w2->>n_mail: POST /v1/messages HTTP/1.1
  n_mail-->>n_w2: HTTP/1.1 503 Service Unavailable
  Note over n_w2,n_dlq: Eksponensial backoff, jitter va dead-letter jadvali
  n_w2->>n_dlq: 09:41:07 WARN job j_9310 attempt 8/8: 550 mailbox does not exist
  Note over n_w2,n_db: Hamma aynan 09:00 ni xohlaydi
  n_w2->>n_db: SELECT tenant, count(*) AS due, max(now() - run_at) AS late
  n_db-->>n_w2: t_9 1 480 220 00:14:51
  Note over n_w1,n_db: Soatlik partitsiyalar va har tenant'ga ulush
  n_w1->>n_db: SELECT j.id FROM active_tenants t
  n_db-->>n_w1: 50 rows from 10 tenants, 2.4 ms

D2

direction: right

n_app: "Tenant ilovasi" {
  shape: rectangle
}
n_mail: "Email provayder" {
  shape: cloud
}
n_api: "Scheduler API" {
  shape: rectangle
}
n_db: "Jobs jadvali" {
  shape: cylinder
}
n_w1: "Worker 1" {
  shape: rectangle
  style.multiple: true
}
n_w2: "Worker 2" {
  shape: rectangle
  style.multiple: true
}
n_dlq: "O'lik job'lar" {
  shape: cylinder
}
n_app2: "Katta tenant" {
  shape: rectangle
}

n_app -> n_api
n_api -> n_db
n_w1 -> n_db
n_w2 -> n_db
n_w1 -> n_mail
n_w2 -> n_mail
n_w2 -> n_dlq {
  target-arrowhead.shape: cross
}
n_app2 -> n_api

PlantUML: so'rovlar ketma-ketligi

@startuml
participant "Tenant ilovasi" as n_app
participant "Cron server A" as n_cron
participant "Cron server B" as n_cron2
boundary "Email provayder" as n_mail
participant "Scheduler API" as n_api
database "Jobs jadvali" as n_db
participant "Worker 1" as n_w1
participant "Worker 2" as n_w2
database "O'lik job'lar" as n_dlq
note over n_app, n_cron : Bitta mashina, bitta crontab
n_app -> n_cron : $ crontab -l # on cron-a
note over n_cron2, n_mail : Server o'ladi, keyin ikki marta qaytadi
n_cron2 -> n_mail : [dublikat] 08:12 cron-a unreachable, jobs 08:15..08:50 never ran
note over n_api, n_db : Jadval vaqti yozilgan qatorlarga aylanadi
n_api -> n_db : INSERT INTO jobs -- the row, as JSON
n_db --> n_api : INSERT 0 1
note over n_w1, n_db : Qatorlarni FOR UPDATE SKIP LOCKED bilan olish
n_w1 -> n_db : WITH picked AS (
n_db --> n_w1 : 50 rows, 3.1 ms
note over n_w1, n_db : O'lgan worker o'z job'larini osilib qoldiradi
n_w1 -> n_db : SELECT id, locked_by, now() - updated_at AS stuck
n_db --> n_w1 : j_8840 | w-2 | 00:47:12
note over n_w1, n_db : Yangilanmasa muddati tugaydigan lease
n_w1 -> n_db : UPDATE jobs
n_db --> n_w1 : lease_until = 09:00:41+05
note over n_w1, n_mail : Har ishga tushirish idempotency kaliti bilan
n_w1 -> n_mail : POST /v1/messages HTTP/1.1
n_mail --> n_w1 : HTTP/1.1 200 OK
note over n_w2, n_mail : Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi
n_w2 -> n_mail : [qayta urinish] POST /v1/messages HTTP/1.1
n_mail --> n_w2 : HTTP/1.1 503 Service Unavailable
note over n_w2, n_dlq : Eksponensial backoff, jitter va dead-letter jadvali
n_w2 -> n_dlq : 09:41:07 WARN job j_9310 attempt 8/8: 550 mailbox does not exist
note over n_w2, n_db : Hamma aynan 09:00 ni xohlaydi
n_w2 -> n_db : SELECT tenant, count(*) AS due, max(now() - run_at) AS late
n_db --> n_w2 : t_9 1 480 220 00:14:51
note over n_w1, n_db : Soatlik partitsiyalar va har tenant'ga ulush
n_w1 -> n_db : SELECT j.id FROM active_tenants t
n_db --> n_w1 : 50 rows from 10 tenants, 2.4 ms
@enduml

PlantUML: komponentlar

@startuml
left to right direction
agent "Tenant ilovasi" as n_app
cloud "Email provayder" as n_mail
component "Scheduler API" as n_api
database "Jobs jadvali" as n_db
collections "Worker 1" as n_w1
collections "Worker 2" as n_w2
database "O'lik job'lar" as n_dlq
agent "Katta tenant" as n_app2
n_app --> n_api
n_api --> n_db
n_w1 --> n_db
n_w2 --> n_db
n_w1 --> n_mail
n_w2 --> n_mail
n_w2 -[#c62828]-> n_dlq
n_app2 --> n_api
@enduml

Xulosa

Scheduler bu vaqtlar bazasi va tirik qolishiga ishonib bo'lmaydigan worker'lar.

1) Bitta cron server o'chiqligida job'larni o'tkazib yuboradi, failover'dan keyin ikki marta bajaradi. 2) Job'larni run_at UTC da yozilgan qatorlar sifatida saqlang; worker'lar FOR UPDATE SKIP LOCKED bilan oladi. 3) Olish heartbeat bilan yangilanadigan lease, baza soati bo'yicha; muddat tugashi osilib qolgan job'larni qaytaradi. 4) Bu ishga tushirishni at least once qiladi, shuning uchun har tashqi ta'sir idempotency kaliti bilan boradi. 5) Eksponensial backoff va jitter bilan qayta rejalashtirib qayta urining; qolganini dead-letter jadvaliga qo'ying va ogohlantirish qo'ying. 6) Vaqt bo'yicha partitsiyalang, eski partitsiyalarni o'chiring va 09:00 adolatli bo'lishi uchun har tenant'dan ulush oling.

Ko'rsatkichlar: boshida va oxirida

Ko'rsatkichBoshidaOxirida
Boshlanish kechikishi p991 s3 s
Bajarilmagan job0/kun0/kun
Ikki marta bajarilgan0/kun0/kun

Qadamlar

  1. Holat Bitta mashina, bitta crontab

    Tenant'lar ilova orqali ish rejalashtiradi: har kuni 09:00 da dayjest, har besh daqiqada kurs sinxroni, ertaga bir martalik eslatma.

  2. Muammo Server o'ladi, keyin ikki marta qaytadi

    08:12 da cron-a tarmoqni yo'qotadi.

  3. Yechim Jadval vaqti yozilgan qatorlarga aylanadi

    Endi API har job'ni qator qilib qo'shadi: nima bajariladi, qaysi tenant uchun va run_at, UTC da.

  4. Yechim Qatorlarni FOR UPDATE SKIP LOCKED bilan olish

    Har worker bitta so'rovda bir to'plamni oladi.

  5. Muammo O'lgan worker o'z job'larini osilib qoldiradi

    Worker 2 09:00:05 da 50 ta job olgan holda o'ldiriladi.

  6. Yechim Yangilanmasa muddati tugaydigan lease

    Endi olish lease beradi: lease_until 30 soniya oldinda.

  7. Yechim Har ishga tushirish idempotency kaliti bilan

    Worker har emailni job id va rejalashtirilgan vaqtdan tuzilgan kalit bilan yuboradi: j_8812:2026-11-19T04:00Z.

  8. Muammo Provayder yiqiladi, worker'lar uni so'rovlar bilan ko'mib tashlaydi

    09:00 da email provayder o'n daqiqa davomida 503 qaytara boshlaydi.

  9. Yechim Eksponensial backoff, jitter va dead-letter jadvali

    Muvaffaqiyatsiz job run_at surilgan holda queued ga qaytadi: 30 s, 1 daqiqa, 2 daqiqa, 32 daqiqagacha ikki baravardan, har kechikish 20% gacha tasodifiy siljiydi.

  10. Muammo Hamma aynan 09:00 ni xohlaydi

    Ko'pchilik yaxlit vaqtni tanlaydi, bitta katta tenant t_9 esa 09:00 ga 1.5 million eslatma rejalashtirgan.

  11. Yechim Soatlik partitsiyalar va har tenant'ga ulush

    Jobs jadvali run_at soati bo'yicha partitsiyalangan.

  12. Xulosa Xulosa

    Scheduler bu vaqtlar bazasi va tirik qolishiga ishonib bo'lmaydigan worker'lar.

Atamalar

SKIP LOCKED
FOR UPDATE ga qo'shimcha: boshqa tranzaksiya qulflagan qatorlarni kutmasdan o'tkazib yuboradi, worker'lar bir-birini to'smaydi.
Lease (ijara muddati)
Ishga vaqtincha egalik huquqi; egasi heartbeat bilan yangilab turmasa, muddati tugaydi va ish boshqasiga o'tadi.
Vaqt bo'yicha partitsiya
Jadvalni soat yoki kun bo'yicha bo'laklarga ajratish; so'rovlar faqat joriy bo'lakka tegadi, eskisi butunlay o'chiriladi.