mufassall

Cheat sheet · Scaling

Caching

13 min · Intermediate · mufassall.dtwin.uz/en/l/caching/

← Open the lesson

The problem

A product page reads the same row from the database thousands of times a second. A cache in front of it makes reads fast, then brings its own problems: stale data, a stampede on a hot key and the question of where writes go.

What you learn

  • The cache-aside read path: look in the cache, fall back to the database, fill
  • TTL and invalidation: why a write must delete the cached copy
  • What a stampede on a hot key is, and how coalescing and jittered TTL stop it
  • Write-through vs write-back, and what each one risks
Final scene
50%30%20%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_client(["Shop app"])
  n_api["Product API"]
  n_db[("Database")]
  n_cache{{"Cache (Redis)"}}
  n_admin(("Shop manager"))
  n_worker("Flush worker")
  n_client --> n_api
  n_api --> n_db
  n_api --> n_cache
  n_admin --> n_api
  n_worker --> n_cache
  n_worker --> n_db

Mermaid: request sequence

sequenceDiagram
  participant n_api as Product API
  participant n_db as Database
  participant n_cache as Cache (Redis)
  Note over n_api,n_db: Every page view goes to the database
  n_api->>n_db: SELECT p.id, p.name, p.price, s.qty
  n_db-->>n_api: 1 row in 38 ms
  Note over n_api,n_cache: Look in the cache first
  n_api->>n_cache: GET product:1042
  n_cache-->>n_api: (nil)
  Note over n_api,n_cache: After a miss, put the answer in the cache
  n_api->>n_cache: SET product:1042 '{...}' EX 300
  n_cache-->>n_api: OK
  Note over n_api,n_cache: A hit: answer from memory
  n_api->>n_cache: GET product:1042
  n_cache-->>n_api: (bulk string, 52 bytes)
  Note over n_api,n_db: The price changed, the cache did not
  n_api->>n_db: UPDATE products SET price = 4490000
  n_db-->>n_api: UPDATE 1
  Note over n_api,n_cache: A write deletes the cached copy
  n_api->>n_cache: DEL product:1042
  n_cache-->>n_api: (integer) 1
  Note over n_api,n_db: A hot key expires during a sale
  n_api->>n_db: 08:00:00.000 key product:1042 expired
  Note over n_api,n_cache: Only one request rebuilds the key
  n_api->>n_cache: SET lock:product:1042 req_7 NX PX 2000
  n_cache-->>n_api: OK
  Note over n_api,n_cache: Do not let keys expire together
  n_api->>n_cache: SET product:1042 '{...}' EX 347
  n_cache-->>n_api: OK
  Note over n_api,n_cache: Writes: through the cache or behind it
  n_api->>n_cache: INCR views:1042
  n_cache-->>n_api: (integer) 58213

D2

direction: right

n_client: "Shop app" {
  shape: rectangle
}
n_api: "Product API" {
  shape: rectangle
}
n_db: "Database" {
  shape: cylinder
}
n_cache: "Cache (Redis)" {
  shape: hexagon
}
n_admin: "Shop manager" {
  shape: person
}
n_worker: "Flush worker" {
  shape: rectangle
  style.multiple: true
}

n_client -> n_api
n_api -> n_db
n_api -> n_cache
n_admin -> n_api
n_worker -> n_cache
n_worker -> n_db

PlantUML: request sequence

@startuml
participant "Product API" as n_api
database "Database" as n_db
collections "Cache (Redis)" as n_cache
note over n_api, n_db : Every page view goes to the database
n_api -> n_db : SELECT p.id, p.name, p.price, s.qty
n_db --> n_api : 1 row in 38 ms
note over n_api, n_cache : Look in the cache first
n_api -> n_cache : GET product:1042
n_cache --> n_api : (nil)
note over n_api, n_cache : After a miss, put the answer in the cache
n_api -> n_cache : SET product:1042 '{...}' EX 300
n_cache --> n_api : OK
note over n_api, n_cache : A hit: answer from memory
n_api -> n_cache : GET product:1042
n_cache --> n_api : (bulk string, 52 bytes)
note over n_api, n_db : The price changed, the cache did not
n_api -> n_db : UPDATE products SET price = 4490000
n_db --> n_api : UPDATE 1
note over n_api, n_cache : A write deletes the cached copy
n_api -> n_cache : DEL product:1042
n_cache --> n_api : (integer) 1
note over n_api, n_db : A hot key expires during a sale
n_api -> n_db : [duplicate] 08:00:00.000 key product:1042 expired
note over n_api, n_cache : Only one request rebuilds the key
n_api -> n_cache : SET lock:product:1042 req_7 NX PX 2000
n_cache --> n_api : OK
note over n_api, n_cache : Do not let keys expire together
n_api -> n_cache : SET product:1042 '{...}' EX 347
n_cache --> n_api : OK
note over n_api, n_cache : Writes: through the cache or behind it
n_api -> n_cache : INCR views:1042
n_cache --> n_api : (integer) 58213
@enduml

PlantUML: components

@startuml
left to right direction
agent "Shop app" as n_client
component "Product API" as n_api
database "Database" as n_db
stack "Cache (Redis)" as n_cache
actor "Shop manager" as n_admin
collections "Flush worker" as n_worker
n_client --> n_api
n_api --> n_db
n_api --> n_cache
n_admin --> n_api
n_worker --> n_cache
n_worker --> n_db
@enduml

Recap

Five rules for a cache.

1) Cache-aside: read the cache, on a miss read the database and fill. 2) Every key has a TTL as a safety net. 3) A write deletes the key, after the commit. 4) Hot keys need coalescing and jittered TTLs. 5) Write-back only for data you can afford to lose.

Numbers: at the start and at the end

MetricStartEnd
Cache hit ratio0%96%
Reply p95850 ms14 ms
Database reads2 400/s110/s
Old price shown for0 s0 s

Steps

  1. Problem Every page view goes to the database

    Each time someone opens product 1042, the API joins two tables and reads the same row again.

  2. Fix Look in the cache first

    A cache sits next to the API.

  3. Fix After a miss, put the answer in the cache

    After reading from the database, the API stores the result in the cache with a time to live (TTL).

  4. Case A hit: answer from memory

    Now most requests find the key.

  5. Problem The price changed, the cache did not

    The manager lowers the price.

  6. Fix A write deletes the cached copy

    After the update commits, the API deletes the key.

  7. Problem A hot key expires during a sale

    At 08:00 a sale starts and thousands of people open the same product.

  8. Fix Only one request rebuilds the key

    On a miss the API first tries to take a short lock.

  9. Fix Do not let keys expire together

    After a deploy, thousands of keys were filled within the same minute with the same TTL, so they all expire within the same minute too.

  10. Case Writes: through the cache or behind it

    So far writes went straight to the database.

  11. Recap Recap

    Five rules for a cache.

Terms

Cache-aside
The most common cache pattern: the app looks in the cache first, on a miss reads the database and puts the answer in the cache. Later requests hit and are served from memory.
TTL (time to live)
How long a cached key lives before it expires by itself. A safety net so stale data cannot stay forever if invalidation is missed.
Cache stampede
When a hot key expires, thousands of requests miss at once and hit the database with the same query. Request coalescing and jittered TTLs prevent it.
Request coalescing
On a miss, only one request takes a short lock, reads the database and refills the cache; the rest wait a few milliseconds and read the fresh key. Thousands of identical queries become one.
Write-through and write-back
Write-through writes the cache and the database in the same request: always fresh, but every write pays twice. Write-back writes only the cache and a worker flushes later: very fast, but unflushed data is lost if the cache dies.