mufassall

Cheat sheet · Shipping

Feature flags

13 min · Intermediate · mufassall.dtwin.uz/en/l/feature-flags/

← Open the lesson

The problem

A shop ships a new checkout flow. When deploy and release are the same act, a bug reaches every user and the way back is a 15-minute redeploy. Feature flags separate the two: the code ships dark, a flag service decides who sees it, the SDK evaluates locally and gets changes over a stream, a kill switch turns it off in two seconds, and owners with expiry dates keep old flags from piling up.

What you learn

  • Why deploy is not release, and what a flag rule with segments and a percentage looks like
  • Local evaluation from a cached snapshot plus streaming updates instead of a remote call per request
  • Percentage rollout by a stable hash of the user id, a kill switch and a guardrail that rolls back by itself
  • Stale flags as debt: owners, expiry dates and a CI check that fails the build
Final scene
8%45%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_users(["Shoppers"])
  n_api["Checkout API"]
  n_db[("Orders DB")]
  n_ci("CI/CD")
  n_flags["Flag service"]
  n_team(("Checkout team"))
  n_users --> n_api
  n_api --> n_db
  n_ci --> n_api
  n_team --> n_flags
  n_api --> n_flags
  n_flags -->|"async"| n_api
  n_ci --x n_flags

Mermaid: request sequence

sequenceDiagram
  participant n_ci as CI/CD
  participant n_api as Checkout API
  participant n_users as Shoppers
  actor n_team as Checkout team
  participant n_flags as Flag service
  participant n_guard as Guardrail
  Note over n_ci,n_api: Deploy and release are the same moment
  n_ci->>n_api: 14:00:00 deploy checkout v412 -#62; 24/24 pods
  Note over n_users,n_api: A bug in the new flow hits everyone
  n_users->>n_api: POST /v1/checkout HTTP/1.1
  n_api-->>n_users: HTTP/1.1 500 Internal Server Error
  Note over n_team,n_flags: The code ships dark, a flag decides
  n_team->>n_flags: PUT /v1/flags/new-checkout HTTP/1.1
  n_flags-->>n_team: HTTP/1.1 200 OK
  Note over n_api,n_flags: A network call per request: slow and fragile
  n_api->>n_flags: 14:20:01 eval new-checkout user=u_42 remote 46 ms
  Note over n_api,n_flags: The SDK keeps a local copy and evaluates in memory
  n_api->>n_flags: GET /v1/sdk/snapshot HTTP/1.1
  n_flags-->>n_api: HTTP/1.1 200 OK
  Note over n_flags,n_api: Changes are pushed over a stream
  n_flags->>n_api: GET /v1/sdk/stream HTTP/1.1
  n_api-->>n_flags: HTTP/1.1 200 OK
  Note over n_users,n_api: Percentage rollout by a stable hash
  n_users->>n_api: eval flag=new-checkout user=u_42 bucket=1873
  Note over n_team,n_flags: An incident: the kill switch, not a rollback
  n_team->>n_flags: PATCH /v1/flags/new-checkout HTTP/1.1
  n_flags-->>n_team: HTTP/1.1 200 OK
  Note over n_guard,n_flags: A guardrail turns the flag off by itself
  n_guard->>n_flags: GUARDRAIL new-checkout rollout 50% window 10 min
  Note over n_ci,n_flags: Old flags pile up into debt
  n_ci->>n_flags: flag-audit 2026-11-18: 142 flags, 61 stale
  Note over n_ci,n_flags: Every flag has an owner and an expiry date
  n_ci->>n_flags: ci flag-lint checkout@v431

D2

direction: right

n_users: "Shoppers" {
  shape: rectangle
}
n_api: "Checkout API" {
  shape: rectangle
}
n_db: "Orders DB" {
  shape: cylinder
}
n_ci: "CI/CD" {
  shape: rectangle
  style.multiple: true
}
n_flags: "Flag service" {
  shape: rectangle
}
n_team: "Checkout team" {
  shape: person
}

n_users -> n_api
n_api -> n_db
n_ci -> n_api
n_team -> n_flags
n_api -> n_flags
n_flags -> n_api: "async"
n_ci -> n_flags {
  target-arrowhead.shape: cross
}

PlantUML: request sequence

@startuml
participant "CI/CD" as n_ci
participant "Checkout API" as n_api
participant "Shoppers" as n_users
actor "Checkout team" as n_team
participant "Flag service" as n_flags
participant "Guardrail" as n_guard
note over n_ci, n_api : Deploy and release are the same moment
n_ci -> n_api : 14:00:00 deploy checkout v412 -> 24/24 pods
note over n_users, n_api : A bug in the new flow hits everyone
n_users -> n_api : POST /v1/checkout HTTP/1.1
n_api --> n_users : HTTP/1.1 500 Internal Server Error
note over n_team, n_flags : The code ships dark, a flag decides
n_team -> n_flags : PUT /v1/flags/new-checkout HTTP/1.1
n_flags --> n_team : HTTP/1.1 200 OK
note over n_api, n_flags : A network call per request: slow and fragile
n_api -> n_flags : 14:20:01 eval new-checkout user=u_42 remote 46 ms
note over n_api, n_flags : The SDK keeps a local copy and evaluates in memory
n_api -> n_flags : GET /v1/sdk/snapshot HTTP/1.1
n_flags --> n_api : HTTP/1.1 200 OK
note over n_flags, n_api : Changes are pushed over a stream
n_flags ->> n_api : GET /v1/sdk/stream HTTP/1.1
n_api -->> n_flags : HTTP/1.1 200 OK
note over n_users, n_api : Percentage rollout by a stable hash
n_users -> n_api : eval flag=new-checkout user=u_42 bucket=1873
note over n_team, n_flags : An incident: the kill switch, not a rollback
n_team -> n_flags : [retry] PATCH /v1/flags/new-checkout HTTP/1.1
n_flags --> n_team : HTTP/1.1 200 OK
note over n_guard, n_flags : A guardrail turns the flag off by itself
n_guard -> n_flags : GUARDRAIL new-checkout rollout 50% window 10 min
note over n_ci, n_flags : Old flags pile up into debt
n_ci -> n_flags : flag-audit 2026-11-18: 142 flags, 61 stale
note over n_ci, n_flags : Every flag has an owner and an expiry date
n_ci -> n_flags : ci flag-lint checkout@v431
@enduml

PlantUML: components

@startuml
left to right direction
agent "Shoppers" as n_users
component "Checkout API" as n_api
database "Orders DB" as n_db
collections "CI/CD" as n_ci
component "Flag service" as n_flags
actor "Checkout team" as n_team
n_users --> n_api
n_api --> n_db
n_ci --> n_api
n_team --> n_flags
n_api --> n_flags
n_flags --> n_api : async
n_ci -[#c62828]-> n_flags
@enduml

Recap

Six rules for feature flags.

1) Deploy is not release: ship code dark and decide who sees it in a flag. 2) Evaluate in memory from an SDK snapshot; never a network call per request, and keep working when the flag service is down. 3) Push changes over a stream so turning off takes seconds. 4) Roll out by a stable hash of the user id so users do not flip and percentages only add people. 5) Keep a tested kill switch and guardrails that roll back on metrics by themselves. 6) Every flag has an owner and an expiry, and CI enforces it.

Numbers: at the start and at the end

MetricStartEnd
Checkout errors0.4%0.4%
Time to turn off900 s2 s
Checkout p99180 ms181 ms
Stale flags04

Steps

  1. Case Deploy and release are the same moment

    The team finishes a new checkout flow and merges the PR.

  2. Problem A bug in the new flow hits everyone

    The new flow rounds tax differently, and some carts fail with 500.

  3. Fix The code ships dark, a flag decides

    The new flow now sits behind if (flags.on("new-checkout", user)).

  4. Problem A network call per request: slow and fragile

    Each checkout now waits for a call to the flag service: 46 to 51 ms, and p99 grows from 180 to 226 ms.

  5. Fix The SDK keeps a local copy and evaluates in memory

    The API now uses the flag SDK.

  6. Fix Changes are pushed over a stream

    The SDK keeps one long-lived connection to the flag service using SSE (server-sent events).

  7. Fix Percentage rollout by a stable hash

    The rollout grows from 5% to 25%.

  8. Fix An incident: the kill switch, not a rollback

    At 25%, card declines in the new flow jump: checkout errors reach 4.8% among those users.

  9. Fix A guardrail turns the flag off by itself

    Every checkout emits an event with the flag value it got.

  10. Problem Old flags pile up into debt

    A year later there are 142 flags, and 61 of them are stale.

  11. Fix Every flag has an owner and an expiry date

    Creating a flag now requires an owner and an expiry date.

  12. Recap Recap

    Six rules for feature flags.

Terms

Kill switch
A flag that turns a feature off for everyone within seconds, without a deploy.
Percentage rollout
A feature turned on for a share of users by a stable hash of the user id, so each user always gets the same answer.
Guardrail metric
A metric compared between flag-on and flag-off users during a rollout; crossing its limit turns the flag off automatically.