Cheat sheet · Scaling
Caching
The problem
A product page reads the same row from the database thousands of times a second. A cache in front of it makes reads fast, then brings its own problems: stale data, a stampede on a hot key and the question of where writes go.
What you learn
- The cache-aside read path: look in the cache, fall back to the database, fill
- TTL and invalidation: why a write must delete the cached copy
- What a stampede on a hot key is, and how coalescing and jittered TTL stop it
- Write-through vs write-back, and what each one risks
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_client(["Shop app"])
n_api["Product API"]
n_db[("Database")]
n_cache{{"Cache (Redis)"}}
n_admin(("Shop manager"))
n_worker("Flush worker")
n_client --> n_api
n_api --> n_db
n_api --> n_cache
n_admin --> n_api
n_worker --> n_cache
n_worker --> n_dbMermaid: request sequence
sequenceDiagram
participant n_api as Product API
participant n_db as Database
participant n_cache as Cache (Redis)
Note over n_api,n_db: Every page view goes to the database
n_api->>n_db: SELECT p.id, p.name, p.price, s.qty
n_db-->>n_api: 1 row in 38 ms
Note over n_api,n_cache: Look in the cache first
n_api->>n_cache: GET product:1042
n_cache-->>n_api: (nil)
Note over n_api,n_cache: After a miss, put the answer in the cache
n_api->>n_cache: SET product:1042 '{...}' EX 300
n_cache-->>n_api: OK
Note over n_api,n_cache: A hit: answer from memory
n_api->>n_cache: GET product:1042
n_cache-->>n_api: (bulk string, 52 bytes)
Note over n_api,n_db: The price changed, the cache did not
n_api->>n_db: UPDATE products SET price = 4490000
n_db-->>n_api: UPDATE 1
Note over n_api,n_cache: A write deletes the cached copy
n_api->>n_cache: DEL product:1042
n_cache-->>n_api: (integer) 1
Note over n_api,n_db: A hot key expires during a sale
n_api->>n_db: 08:00:00.000 key product:1042 expired
Note over n_api,n_cache: Only one request rebuilds the key
n_api->>n_cache: SET lock:product:1042 req_7 NX PX 2000
n_cache-->>n_api: OK
Note over n_api,n_cache: Do not let keys expire together
n_api->>n_cache: SET product:1042 '{...}' EX 347
n_cache-->>n_api: OK
Note over n_api,n_cache: Writes: through the cache or behind it
n_api->>n_cache: INCR views:1042
n_cache-->>n_api: (integer) 58213D2
direction: right
n_client: "Shop app" {
shape: rectangle
}
n_api: "Product API" {
shape: rectangle
}
n_db: "Database" {
shape: cylinder
}
n_cache: "Cache (Redis)" {
shape: hexagon
}
n_admin: "Shop manager" {
shape: person
}
n_worker: "Flush worker" {
shape: rectangle
style.multiple: true
}
n_client -> n_api
n_api -> n_db
n_api -> n_cache
n_admin -> n_api
n_worker -> n_cache
n_worker -> n_dbPlantUML: request sequence
@startuml
participant "Product API" as n_api
database "Database" as n_db
collections "Cache (Redis)" as n_cache
note over n_api, n_db : Every page view goes to the database
n_api -> n_db : SELECT p.id, p.name, p.price, s.qty
n_db --> n_api : 1 row in 38 ms
note over n_api, n_cache : Look in the cache first
n_api -> n_cache : GET product:1042
n_cache --> n_api : (nil)
note over n_api, n_cache : After a miss, put the answer in the cache
n_api -> n_cache : SET product:1042 '{...}' EX 300
n_cache --> n_api : OK
note over n_api, n_cache : A hit: answer from memory
n_api -> n_cache : GET product:1042
n_cache --> n_api : (bulk string, 52 bytes)
note over n_api, n_db : The price changed, the cache did not
n_api -> n_db : UPDATE products SET price = 4490000
n_db --> n_api : UPDATE 1
note over n_api, n_cache : A write deletes the cached copy
n_api -> n_cache : DEL product:1042
n_cache --> n_api : (integer) 1
note over n_api, n_db : A hot key expires during a sale
n_api -> n_db : [duplicate] 08:00:00.000 key product:1042 expired
note over n_api, n_cache : Only one request rebuilds the key
n_api -> n_cache : SET lock:product:1042 req_7 NX PX 2000
n_cache --> n_api : OK
note over n_api, n_cache : Do not let keys expire together
n_api -> n_cache : SET product:1042 '{...}' EX 347
n_cache --> n_api : OK
note over n_api, n_cache : Writes: through the cache or behind it
n_api -> n_cache : INCR views:1042
n_cache --> n_api : (integer) 58213
@endumlPlantUML: components
@startuml
left to right direction
agent "Shop app" as n_client
component "Product API" as n_api
database "Database" as n_db
stack "Cache (Redis)" as n_cache
actor "Shop manager" as n_admin
collections "Flush worker" as n_worker
n_client --> n_api
n_api --> n_db
n_api --> n_cache
n_admin --> n_api
n_worker --> n_cache
n_worker --> n_db
@endumlRecap
Five rules for a cache.
1) Cache-aside: read the cache, on a miss read the database and fill. 2) Every key has a TTL as a safety net. 3) A write deletes the key, after the commit. 4) Hot keys need coalescing and jittered TTLs. 5) Write-back only for data you can afford to lose.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Cache hit ratio | 0% | 96% |
| Reply p95 | 850 ms | 14 ms |
| Database reads | 2 400/s | 110/s |
| Old price shown for | 0 s | 0 s |
Steps
- Problem Every page view goes to the database
Each time someone opens product
1042, the API joins two tables and reads the same row again. - Fix Look in the cache first
A cache sits next to the API.
- Fix After a miss, put the answer in the cache
After reading from the database, the API stores the result in the cache with a time to live (TTL).
- Case A hit: answer from memory
Now most requests find the key.
- Problem The price changed, the cache did not
The manager lowers the price.
- Fix A write deletes the cached copy
After the update commits, the API deletes the key.
- Problem A hot key expires during a sale
At 08:00 a sale starts and thousands of people open the same product.
- Fix Only one request rebuilds the key
On a miss the API first tries to take a short lock.
- Fix Do not let keys expire together
After a deploy, thousands of keys were filled within the same minute with the same TTL, so they all expire within the same minute too.
- Case Writes: through the cache or behind it
So far writes went straight to the database.
- Recap Recap
Five rules for a cache.
Terms
- Cache-aside
- The most common cache pattern: the app looks in the cache first, on a miss reads the database and puts the answer in the cache. Later requests hit and are served from memory.
- TTL (time to live)
- How long a cached key lives before it expires by itself. A safety net so stale data cannot stay forever if invalidation is missed.
- Cache stampede
- When a hot key expires, thousands of requests miss at once and hit the database with the same query. Request coalescing and jittered TTLs prevent it.
- Request coalescing
- On a miss, only one request takes a short lock, reads the database and refills the cache; the rest wait a few milliseconds and read the fresh key. Thousands of identical queries become one.
- Write-through and write-back
- Write-through writes the cache and the database in the same request: always fresh, but every write pays twice. Write-back writes only the cache and a worker flushes later: very fast, but unflushed data is lost if the cache dies.