Cheat sheet · Architecture
Distributed locks
The problem
A marketplace pays its sellers with a nightly job. With two pods for availability the job runs twice and every seller is paid twice. A lock in Redis makes one pod wait for the other, until a lease runs out mid-job or a paused process wakes up still believing it holds the lock. Fencing tokens checked by the database, an idempotency key at the bank and an honest look at Redlock show what a lock can and cannot promise, and when a lock in the database itself is the better tool.
What you learn
- Why a mutex inside one process does nothing across pods, and how
SET key token NX PXbuilds a lock with a lease - Releasing and renewing only your own lock, with a token compared inside a Lua script
- Why a GC pause or a clock jump breaks any lease, and how fencing tokens checked by the storage make a stale holder harmless
- What Redlock does and does not fix, and when a database lock or leader election is the better choice
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_a("Worker A")
n_db[("Payouts DB")]
n_bank>"Bank"]
n_b("Worker B")
n_a --> n_db
n_a -->|"call"| n_bank
n_b --> n_db
n_b -->|"call"| n_bankMermaid: request sequence
sequenceDiagram
participant n_a as Worker A
participant n_bank as Bank
participant n_b as Worker B
participant n_redis as Redis
participant n_db as Payouts DB
Note over n_a,n_bank: One pod pays the sellers every night
n_a->>n_bank: 02:00:00 pod-a cron settle-payouts batch b_1118
Note over n_b,n_bank: A second pod, and every seller is paid twice
n_b->>n_bank: 02:00:00.004 pod-b cron settle-payouts batch b_1118
Note over n_a,n_redis: A lock in Redis: SET only if absent, with a lease
n_a->>n_redis: SET lock:payouts:2026-11-18 7f3c9a1e NX PX 30000
n_redis-->>n_a: OK
Note over n_a,n_redis: Release only the lock that is still yours
n_a->>n_redis: EVAL release.lua 1 lock:payouts:2026-11-18 7f3c9a1e
n_redis-->>n_a: (integer) 1
Note over n_b,n_redis: The job runs longer than the lease
n_b->>n_redis: 02:00:00 pod-a lock OK, lease 30 s, month end: 8 000 payouts
Note over n_a,n_redis: Renew the lease while the job runs
n_a->>n_redis: EVAL renew.lua 1 lock:payouts:2026-11-18 7f3c9a1e 30000
n_redis-->>n_a: (integer) 1
Note over n_a,n_bank: The holder freezes past its lease and wakes up
n_a->>n_bank: 02:00:05.000 pod-a lock OK, claims payout po_5501
Note over n_a,n_db: Fencing tokens: the database rejects the old holder
n_a->>n_db: UPDATE payouts SET status = 'paying', fence = 33
n_db-->>n_a: UPDATE 0
Note over n_a,n_bank: The bank cannot check fences: use an idempotency key
n_a->>n_bank: POST /v1/transfers HTTP/1.1
n_bank-->>n_a: HTTP/1.1 200 OK
Note over n_b,n_redis: A Redis failover hands out the same lock twice
n_b->>n_redis: 02:00:00.000 pod-a SET NX on the Redis primary -#62; OK
Note over n_a,n_db: Lock in the database that holds the data
n_a->>n_db: BEGIN#59;
n_db-->>n_a: pg_try_advisory_xact_lock = trueD2
direction: right
n_a: "Worker A" {
shape: rectangle
style.multiple: true
}
n_db: "Payouts DB" {
shape: cylinder
}
n_bank: "Bank" {
shape: cloud
}
n_b: "Worker B" {
shape: rectangle
style.multiple: true
}
n_a -> n_db
n_a -> n_bank: "call"
n_b -> n_db
n_b -> n_bank: "call"PlantUML: request sequence
@startuml
participant "Worker A" as n_a
boundary "Bank" as n_bank
participant "Worker B" as n_b
collections "Redis" as n_redis
database "Payouts DB" as n_db
note over n_a, n_bank : One pod pays the sellers every night
n_a -> n_bank : 02:00:00 pod-a cron settle-payouts batch b_1118
note over n_b, n_bank : A second pod, and every seller is paid twice
n_b -> n_bank : [duplicate] 02:00:00.004 pod-b cron settle-payouts batch b_1118
note over n_a, n_redis : A lock in Redis: SET only if absent, with a lease
n_a -> n_redis : SET lock:payouts:2026-11-18 7f3c9a1e NX PX 30000
n_redis --> n_a : OK
note over n_a, n_redis : Release only the lock that is still yours
n_a -> n_redis : EVAL release.lua 1 lock:payouts:2026-11-18 7f3c9a1e
n_redis --> n_a : (integer) 1
note over n_b, n_redis : The job runs longer than the lease
n_b -> n_redis : 02:00:00 pod-a lock OK, lease 30 s, month end: 8 000 payouts
note over n_a, n_redis : Renew the lease while the job runs
n_a -> n_redis : EVAL renew.lua 1 lock:payouts:2026-11-18 7f3c9a1e 30000
n_redis --> n_a : (integer) 1
note over n_a, n_bank : The holder freezes past its lease and wakes up
n_a -> n_bank : [duplicate] 02:00:05.000 pod-a lock OK, claims payout po_5501
note over n_a, n_db : Fencing tokens: the database rejects the old holder
n_a -> n_db : UPDATE payouts SET status = 'paying', fence = 33
n_db --> n_a : UPDATE 0
note over n_a, n_bank : The bank cannot check fences: use an idempotency key
n_a -> n_bank : POST /v1/transfers HTTP/1.1
n_bank --> n_a : HTTP/1.1 200 OK
note over n_b, n_redis : A Redis failover hands out the same lock twice
n_b -> n_redis : 02:00:00.000 pod-a SET NX on the Redis primary -> OK
note over n_a, n_db : Lock in the database that holds the data
n_a -> n_db : BEGIN;
n_db --> n_a : pg_try_advisory_xact_lock = true
@endumlPlantUML: components
@startuml
left to right direction
collections "Worker A" as n_a
database "Payouts DB" as n_db
cloud "Bank" as n_bank
collections "Worker B" as n_b
n_a --> n_db
n_a --> n_bank : call
n_b --> n_db
n_b --> n_bank : call
@endumlRecap
Six rules for distributed locks.
1) A mutex protects one process; across pods the lock must live in a shared place. 2) Lock with one atomic SET key token NX PX, and release or renew only with a token check in a script. 3) Renew the lease while working, and stop at once when renewal fails. 4) Any lease can be outlived by a pause or a clock jump, so the storage must check a fencing token on every write. 5) Where fences cannot be checked, such as an external API, use idempotency keys. 6) Prefer a lock in the database that holds the data, or leader election on a consensus store; treat a Redis lock, Redlock included, as an efficiency tool.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Double payouts | 0/day | 0/day |
| Stale writes accepted | 0 | 0 |
| Stale writes rejected | 0 | 1 |
Steps
- Case One pod pays the sellers every night
At 02:00 a scheduled job reads the pending payouts, sends a transfer to the bank for each one and marks it paid.
- Problem A second pod, and every seller is paid twice
For availability the service now runs on two pods.
- Fix A lock in Redis: SET only if absent, with a lease
Before the run, each pod tries
SET lock:payouts:<date> <token> NX PX 30000. - Fix Release only the lock that is still yours
When the job ends, the pod deletes the key.
- Problem The job runs longer than the lease
Month end: 8 000 payouts, and the job takes 45 seconds.
- Fix Renew the lease while the job runs
A watchdog thread in the job extends the lease every 10 seconds, with the same token check:
PEXPIREonly if the key is still ours. - Problem The holder freezes past its lease and wakes up
At 02:00:05 pod-a claims payout
po_5501, and the runtime stops the whole process for a full garbage collection. - Fix Fencing tokens: the database rejects the old holder
The lock script now also runs
INCRon a counter and returns the number with the lock: pod-a got 33, pod-b later got 34. - Fix The bank cannot check fences: use an idempotency key
The bank knows nothing about fencing tokens.
- Case A Redis failover hands out the same lock twice
Redis replicates asynchronously.
- Fix Lock in the database that holds the data
The payouts live in one Postgres anyway, so the job takes its lock there.
- Recap Recap
Six rules for distributed locks.
Terms
- Distributed lock
- A lock kept in a shared place (Redis, a database, a consensus store) so that only one of several processes or machines does the work at a time. It usually comes with a lease, so on its own it does not guarantee correctness.
- Lock token (owner value)
- A random value written into the key when the lock is taken. Release and renewal only happen if the key still holds that token, so a holder whose lease expired cannot delete someone else's lock.
- Redlock
- An algorithm that asks 5 independent Redis nodes for the lock and counts it as taken only if a majority grants it within the lease. It survives one node failing, but gives no fencing token and assumes bounded pauses and clock drift.
- Advisory lock
- A lock the database holds on any key the application chooses, such as `pg_try_advisory_xact_lock` in Postgres. It ends with the transaction or the session, so no separate lock service is needed.
- Process pause (stop-the-world)
- The whole process stopping for seconds because of garbage collection, a VM migration or swapping. The process does not notice, so it can carry on as if it held the lock after its lease has run out.