Cheat sheet · Shipping
Feature flags
The problem
A shop ships a new checkout flow. When deploy and release are the same act, a bug reaches every user and the way back is a 15-minute redeploy. Feature flags separate the two: the code ships dark, a flag service decides who sees it, the SDK evaluates locally and gets changes over a stream, a kill switch turns it off in two seconds, and owners with expiry dates keep old flags from piling up.
What you learn
- Why deploy is not release, and what a flag rule with segments and a percentage looks like
- Local evaluation from a cached snapshot plus streaming updates instead of a remote call per request
- Percentage rollout by a stable hash of the user id, a kill switch and a guardrail that rolls back by itself
- Stale flags as debt: owners, expiry dates and a CI check that fails the build
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_users(["Shoppers"])
n_api["Checkout API"]
n_db[("Orders DB")]
n_ci("CI/CD")
n_flags["Flag service"]
n_team(("Checkout team"))
n_users --> n_api
n_api --> n_db
n_ci --> n_api
n_team --> n_flags
n_api --> n_flags
n_flags -->|"async"| n_api
n_ci --x n_flagsMermaid: request sequence
sequenceDiagram
participant n_ci as CI/CD
participant n_api as Checkout API
participant n_users as Shoppers
actor n_team as Checkout team
participant n_flags as Flag service
participant n_guard as Guardrail
Note over n_ci,n_api: Deploy and release are the same moment
n_ci->>n_api: 14:00:00 deploy checkout v412 -#62; 24/24 pods
Note over n_users,n_api: A bug in the new flow hits everyone
n_users->>n_api: POST /v1/checkout HTTP/1.1
n_api-->>n_users: HTTP/1.1 500 Internal Server Error
Note over n_team,n_flags: The code ships dark, a flag decides
n_team->>n_flags: PUT /v1/flags/new-checkout HTTP/1.1
n_flags-->>n_team: HTTP/1.1 200 OK
Note over n_api,n_flags: A network call per request: slow and fragile
n_api->>n_flags: 14:20:01 eval new-checkout user=u_42 remote 46 ms
Note over n_api,n_flags: The SDK keeps a local copy and evaluates in memory
n_api->>n_flags: GET /v1/sdk/snapshot HTTP/1.1
n_flags-->>n_api: HTTP/1.1 200 OK
Note over n_flags,n_api: Changes are pushed over a stream
n_flags->>n_api: GET /v1/sdk/stream HTTP/1.1
n_api-->>n_flags: HTTP/1.1 200 OK
Note over n_users,n_api: Percentage rollout by a stable hash
n_users->>n_api: eval flag=new-checkout user=u_42 bucket=1873
Note over n_team,n_flags: An incident: the kill switch, not a rollback
n_team->>n_flags: PATCH /v1/flags/new-checkout HTTP/1.1
n_flags-->>n_team: HTTP/1.1 200 OK
Note over n_guard,n_flags: A guardrail turns the flag off by itself
n_guard->>n_flags: GUARDRAIL new-checkout rollout 50% window 10 min
Note over n_ci,n_flags: Old flags pile up into debt
n_ci->>n_flags: flag-audit 2026-11-18: 142 flags, 61 stale
Note over n_ci,n_flags: Every flag has an owner and an expiry date
n_ci->>n_flags: ci flag-lint checkout@v431D2
direction: right
n_users: "Shoppers" {
shape: rectangle
}
n_api: "Checkout API" {
shape: rectangle
}
n_db: "Orders DB" {
shape: cylinder
}
n_ci: "CI/CD" {
shape: rectangle
style.multiple: true
}
n_flags: "Flag service" {
shape: rectangle
}
n_team: "Checkout team" {
shape: person
}
n_users -> n_api
n_api -> n_db
n_ci -> n_api
n_team -> n_flags
n_api -> n_flags
n_flags -> n_api: "async"
n_ci -> n_flags {
target-arrowhead.shape: cross
}PlantUML: request sequence
@startuml
participant "CI/CD" as n_ci
participant "Checkout API" as n_api
participant "Shoppers" as n_users
actor "Checkout team" as n_team
participant "Flag service" as n_flags
participant "Guardrail" as n_guard
note over n_ci, n_api : Deploy and release are the same moment
n_ci -> n_api : 14:00:00 deploy checkout v412 -> 24/24 pods
note over n_users, n_api : A bug in the new flow hits everyone
n_users -> n_api : POST /v1/checkout HTTP/1.1
n_api --> n_users : HTTP/1.1 500 Internal Server Error
note over n_team, n_flags : The code ships dark, a flag decides
n_team -> n_flags : PUT /v1/flags/new-checkout HTTP/1.1
n_flags --> n_team : HTTP/1.1 200 OK
note over n_api, n_flags : A network call per request: slow and fragile
n_api -> n_flags : 14:20:01 eval new-checkout user=u_42 remote 46 ms
note over n_api, n_flags : The SDK keeps a local copy and evaluates in memory
n_api -> n_flags : GET /v1/sdk/snapshot HTTP/1.1
n_flags --> n_api : HTTP/1.1 200 OK
note over n_flags, n_api : Changes are pushed over a stream
n_flags ->> n_api : GET /v1/sdk/stream HTTP/1.1
n_api -->> n_flags : HTTP/1.1 200 OK
note over n_users, n_api : Percentage rollout by a stable hash
n_users -> n_api : eval flag=new-checkout user=u_42 bucket=1873
note over n_team, n_flags : An incident: the kill switch, not a rollback
n_team -> n_flags : [retry] PATCH /v1/flags/new-checkout HTTP/1.1
n_flags --> n_team : HTTP/1.1 200 OK
note over n_guard, n_flags : A guardrail turns the flag off by itself
n_guard -> n_flags : GUARDRAIL new-checkout rollout 50% window 10 min
note over n_ci, n_flags : Old flags pile up into debt
n_ci -> n_flags : flag-audit 2026-11-18: 142 flags, 61 stale
note over n_ci, n_flags : Every flag has an owner and an expiry date
n_ci -> n_flags : ci flag-lint checkout@v431
@endumlPlantUML: components
@startuml
left to right direction
agent "Shoppers" as n_users
component "Checkout API" as n_api
database "Orders DB" as n_db
collections "CI/CD" as n_ci
component "Flag service" as n_flags
actor "Checkout team" as n_team
n_users --> n_api
n_api --> n_db
n_ci --> n_api
n_team --> n_flags
n_api --> n_flags
n_flags --> n_api : async
n_ci -[#c62828]-> n_flags
@endumlRecap
Six rules for feature flags.
1) Deploy is not release: ship code dark and decide who sees it in a flag. 2) Evaluate in memory from an SDK snapshot; never a network call per request, and keep working when the flag service is down. 3) Push changes over a stream so turning off takes seconds. 4) Roll out by a stable hash of the user id so users do not flip and percentages only add people. 5) Keep a tested kill switch and guardrails that roll back on metrics by themselves. 6) Every flag has an owner and an expiry, and CI enforces it.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Checkout errors | 0.4% | 0.4% |
| Time to turn off | 900 s | 2 s |
| Checkout p99 | 180 ms | 181 ms |
| Stale flags | 0 | 4 |
Steps
- Case Deploy and release are the same moment
The team finishes a new checkout flow and merges the PR.
- Problem A bug in the new flow hits everyone
The new flow rounds tax differently, and some carts fail with
500. - Fix The code ships dark, a flag decides
The new flow now sits behind
if (flags.on("new-checkout", user)). - Problem A network call per request: slow and fragile
Each checkout now waits for a call to the flag service: 46 to 51 ms, and p99 grows from 180 to 226 ms.
- Fix The SDK keeps a local copy and evaluates in memory
The API now uses the flag SDK.
- Fix Changes are pushed over a stream
The SDK keeps one long-lived connection to the flag service using SSE (server-sent events).
- Fix Percentage rollout by a stable hash
The rollout grows from 5% to 25%.
- Fix An incident: the kill switch, not a rollback
At 25%, card declines in the new flow jump: checkout errors reach 4.8% among those users.
- Fix A guardrail turns the flag off by itself
Every checkout emits an event with the flag value it got.
- Problem Old flags pile up into debt
A year later there are 142 flags, and 61 of them are stale.
- Fix Every flag has an owner and an expiry date
Creating a flag now requires an owner and an expiry date.
- Recap Recap
Six rules for feature flags.
Terms
- Kill switch
- A flag that turns a feature off for everyone within seconds, without a deploy.
- Percentage rollout
- A feature turned on for a share of users by a stable hash of the user id, so each user always gets the same answer.
- Guardrail metric
- A metric compared between flag-on and flag-off users during a rollout; crossing its limit turns the flag off automatically.