Whole systems / 7 / 7
Design a job scheduler
"Send the digest at 09:00", "remind me tomorrow", "sync every 5 minutes": millions of delayed and recurring jobs for many tenants. We go from one cron box to a jobs table claimed with `SKIP LOCKED`, leases with heartbeats, idempotent at-least-once runs, retries with backoff and a dead-letter table, then survive the 09:00 spike with time partitions and per-tenant fairness.
- 15 min
- Advanced
- 12 steps
- Start lag p99
- 1 s
- Jobs not run
- 0 /day
- Jobs run twice
- 0 /day
- request
- reply
- redelivery
- duplicate
- lost
Numbers are a model scenario, not measurements
The real request
Tenant app → Cron box ARequest
$ crontab -l # on cron-a
0 9 * * * /opt/jobs/send-digest --tenant t_171
*/5 * * * * /opt/jobs/sync-rates2
# one-off lines appended by the API:
0 9 19 11 * /opt/jobs/remind --user u_42 --once3
0 9 * * *— Minute 0, hour 9, every day. In the server's local time zone, whatever that is.*/5— Every five minutes. If a run takes six, two copies overlap.--once— A one-off job faked as a yearly line. Someone has to delete it after it runs.
Build it yourselfCheat sheetCompare lessons
Course: 74 lessons
01 Foundations
02 Scaling
03 Data & storage
04 Communication
05 Reliability
06 Architecture
07 Security
08 Shipping
09 Whole systems
10 Business processes
11 Case studies