mufassall

Cheat sheet · Architecture

Distributed locks

14 min · Advanced · mufassall.dtwin.uz/en/l/distributed-locks/

← Open the lesson

The problem

A marketplace pays its sellers with a nightly job. With two pods for availability the job runs twice and every seller is paid twice. A lock in Redis makes one pod wait for the other, until a lease runs out mid-job or a paused process wakes up still believing it holds the lock. Fencing tokens checked by the database, an idempotency key at the bank and an honest look at Redlock show what a lock can and cannot promise, and when a lock in the database itself is the better tool.

What you learn

  • Why a mutex inside one process does nothing across pods, and how SET key token NX PX builds a lock with a lease
  • Releasing and renewing only your own lock, with a token compared inside a Lua script
  • Why a GC pause or a clock jump breaks any lease, and how fencing tokens checked by the storage make a stale holder harmless
  • What Redlock does and does not fix, and when a database lock or leader election is the better choice
Final scene

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_a("Worker A")
  n_db[("Payouts DB")]
  n_bank>"Bank"]
  n_b("Worker B")
  n_a --> n_db
  n_a -->|"call"| n_bank
  n_b --> n_db
  n_b -->|"call"| n_bank

Mermaid: request sequence

sequenceDiagram
  participant n_a as Worker A
  participant n_bank as Bank
  participant n_b as Worker B
  participant n_redis as Redis
  participant n_db as Payouts DB
  Note over n_a,n_bank: One pod pays the sellers every night
  n_a->>n_bank: 02:00:00 pod-a cron settle-payouts batch b_1118
  Note over n_b,n_bank: A second pod, and every seller is paid twice
  n_b->>n_bank: 02:00:00.004 pod-b cron settle-payouts batch b_1118
  Note over n_a,n_redis: A lock in Redis: SET only if absent, with a lease
  n_a->>n_redis: SET lock:payouts:2026-11-18 7f3c9a1e NX PX 30000
  n_redis-->>n_a: OK
  Note over n_a,n_redis: Release only the lock that is still yours
  n_a->>n_redis: EVAL release.lua 1 lock:payouts:2026-11-18 7f3c9a1e
  n_redis-->>n_a: (integer) 1
  Note over n_b,n_redis: The job runs longer than the lease
  n_b->>n_redis: 02:00:00 pod-a lock OK, lease 30 s, month end: 8 000 payouts
  Note over n_a,n_redis: Renew the lease while the job runs
  n_a->>n_redis: EVAL renew.lua 1 lock:payouts:2026-11-18 7f3c9a1e 30000
  n_redis-->>n_a: (integer) 1
  Note over n_a,n_bank: The holder freezes past its lease and wakes up
  n_a->>n_bank: 02:00:05.000 pod-a lock OK, claims payout po_5501
  Note over n_a,n_db: Fencing tokens: the database rejects the old holder
  n_a->>n_db: UPDATE payouts SET status = 'paying', fence = 33
  n_db-->>n_a: UPDATE 0
  Note over n_a,n_bank: The bank cannot check fences: use an idempotency key
  n_a->>n_bank: POST /v1/transfers HTTP/1.1
  n_bank-->>n_a: HTTP/1.1 200 OK
  Note over n_b,n_redis: A Redis failover hands out the same lock twice
  n_b->>n_redis: 02:00:00.000 pod-a SET NX on the Redis primary -#62; OK
  Note over n_a,n_db: Lock in the database that holds the data
  n_a->>n_db: BEGIN#59;
  n_db-->>n_a: pg_try_advisory_xact_lock = true

D2

direction: right

n_a: "Worker A" {
  shape: rectangle
  style.multiple: true
}
n_db: "Payouts DB" {
  shape: cylinder
}
n_bank: "Bank" {
  shape: cloud
}
n_b: "Worker B" {
  shape: rectangle
  style.multiple: true
}

n_a -> n_db
n_a -> n_bank: "call"
n_b -> n_db
n_b -> n_bank: "call"

PlantUML: request sequence

@startuml
participant "Worker A" as n_a
boundary "Bank" as n_bank
participant "Worker B" as n_b
collections "Redis" as n_redis
database "Payouts DB" as n_db
note over n_a, n_bank : One pod pays the sellers every night
n_a -> n_bank : 02:00:00 pod-a cron settle-payouts batch b_1118
note over n_b, n_bank : A second pod, and every seller is paid twice
n_b -> n_bank : [duplicate] 02:00:00.004 pod-b cron settle-payouts batch b_1118
note over n_a, n_redis : A lock in Redis: SET only if absent, with a lease
n_a -> n_redis : SET lock:payouts:2026-11-18 7f3c9a1e NX PX 30000
n_redis --> n_a : OK
note over n_a, n_redis : Release only the lock that is still yours
n_a -> n_redis : EVAL release.lua 1 lock:payouts:2026-11-18 7f3c9a1e
n_redis --> n_a : (integer) 1
note over n_b, n_redis : The job runs longer than the lease
n_b -> n_redis : 02:00:00 pod-a lock OK, lease 30 s, month end: 8 000 payouts
note over n_a, n_redis : Renew the lease while the job runs
n_a -> n_redis : EVAL renew.lua 1 lock:payouts:2026-11-18 7f3c9a1e 30000
n_redis --> n_a : (integer) 1
note over n_a, n_bank : The holder freezes past its lease and wakes up
n_a -> n_bank : [duplicate] 02:00:05.000 pod-a lock OK, claims payout po_5501
note over n_a, n_db : Fencing tokens: the database rejects the old holder
n_a -> n_db : UPDATE payouts SET status = 'paying', fence = 33
n_db --> n_a : UPDATE 0
note over n_a, n_bank : The bank cannot check fences: use an idempotency key
n_a -> n_bank : POST /v1/transfers HTTP/1.1
n_bank --> n_a : HTTP/1.1 200 OK
note over n_b, n_redis : A Redis failover hands out the same lock twice
n_b -> n_redis : 02:00:00.000 pod-a SET NX on the Redis primary -> OK
note over n_a, n_db : Lock in the database that holds the data
n_a -> n_db : BEGIN;
n_db --> n_a : pg_try_advisory_xact_lock = true
@enduml

PlantUML: components

@startuml
left to right direction
collections "Worker A" as n_a
database "Payouts DB" as n_db
cloud "Bank" as n_bank
collections "Worker B" as n_b
n_a --> n_db
n_a --> n_bank : call
n_b --> n_db
n_b --> n_bank : call
@enduml

Recap

Six rules for distributed locks.

1) A mutex protects one process; across pods the lock must live in a shared place. 2) Lock with one atomic SET key token NX PX, and release or renew only with a token check in a script. 3) Renew the lease while working, and stop at once when renewal fails. 4) Any lease can be outlived by a pause or a clock jump, so the storage must check a fencing token on every write. 5) Where fences cannot be checked, such as an external API, use idempotency keys. 6) Prefer a lock in the database that holds the data, or leader election on a consensus store; treat a Redis lock, Redlock included, as an efficiency tool.

Numbers: at the start and at the end

MetricStartEnd
Double payouts0/day0/day
Stale writes accepted00
Stale writes rejected01

Steps

  1. Case One pod pays the sellers every night

    At 02:00 a scheduled job reads the pending payouts, sends a transfer to the bank for each one and marks it paid.

  2. Problem A second pod, and every seller is paid twice

    For availability the service now runs on two pods.

  3. Fix A lock in Redis: SET only if absent, with a lease

    Before the run, each pod tries SET lock:payouts:<date> <token> NX PX 30000.

  4. Fix Release only the lock that is still yours

    When the job ends, the pod deletes the key.

  5. Problem The job runs longer than the lease

    Month end: 8 000 payouts, and the job takes 45 seconds.

  6. Fix Renew the lease while the job runs

    A watchdog thread in the job extends the lease every 10 seconds, with the same token check: PEXPIRE only if the key is still ours.

  7. Problem The holder freezes past its lease and wakes up

    At 02:00:05 pod-a claims payout po_5501, and the runtime stops the whole process for a full garbage collection.

  8. Fix Fencing tokens: the database rejects the old holder

    The lock script now also runs INCR on a counter and returns the number with the lock: pod-a got 33, pod-b later got 34.

  9. Fix The bank cannot check fences: use an idempotency key

    The bank knows nothing about fencing tokens.

  10. Case A Redis failover hands out the same lock twice

    Redis replicates asynchronously.

  11. Fix Lock in the database that holds the data

    The payouts live in one Postgres anyway, so the job takes its lock there.

  12. Recap Recap

    Six rules for distributed locks.

Terms

Distributed lock
A lock kept in a shared place (Redis, a database, a consensus store) so that only one of several processes or machines does the work at a time. It usually comes with a lease, so on its own it does not guarantee correctness.
Lock token (owner value)
A random value written into the key when the lock is taken. Release and renewal only happen if the key still holds that token, so a holder whose lease expired cannot delete someone else's lock.
Redlock
An algorithm that asks 5 independent Redis nodes for the lock and counts it as taken only if a majority grants it within the lease. It survives one node failing, but gives no fencing token and assumes bounded pauses and clock drift.
Advisory lock
A lock the database holds on any key the application chooses, such as `pg_try_advisory_xact_lock` in Postgres. It ends with the transaction or the session, so no separate lock service is needed.
Process pause (stop-the-world)
The whole process stopping for seconds because of garbage collection, a VM migration or swapping. The process does not notice, so it can carry on as if it held the lock after its lease has run out.