Cheat sheet · Shipping
CI/CD: from commit to production
The problem
A team deploys once a week, by hand, and every third deploy breaks something. Step by step the path from commit to production becomes automatic, small and reversible, and the team ends up deploying a dozen times a day.
What you learn
- CI builds and tests every commit once and produces one immutable artifact
- Rolling, blue-green and canary deploys: what each costs and how fast each rolls back
- Automatic rollback on error rate, and feature flags that separate deploy from release
- Four delivery numbers: deploys per day, lead time, change failure rate, time to restore
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_dev(("Developer"))
n_users(["Users"])
n_lb{"Load balancer"}
n_ci("CI/CD")
n_registry[("Artifact registry")]
n_green["Green servers"]
n_blue["Blue servers"]
n_flags["Feature flags"]
n_users --> n_lb
n_dev --> n_ci
n_ci --> n_registry
n_ci --> n_green
n_lb --> n_green
n_lb --> n_blue
n_ci --> n_lb
n_green --> n_flagsMermaid: request sequence
sequenceDiagram
actor n_dev as Developer
participant n_blue as Blue servers
participant n_ci as CI/CD
participant n_lb as Load balancer
participant n_green as Green servers
participant n_flags as Feature flags
Note over n_dev,n_blue: Friday deploy, by hand, to six servers
n_dev->>n_blue: $ ssh deploy@192.0.2.12
Note over n_dev,n_ci: CI: every commit builds and tests once
n_dev->>n_ci: commit 3f9a1c2 by u_42: checkout, new card form
Note over n_lb,n_green: Rolling deploy: one server at a time
n_lb->>n_green: GET /healthz HTTP/1.1
n_green-->>n_lb: HTTP/1.1 200 OK
Note over n_lb,n_green: Health checks pass, checkout fails
n_lb->>n_green: POST /checkout HTTP/1.1
n_green-->>n_lb: HTTP/1.1 500 Internal Server Error
Note over n_ci,n_lb: Blue-green: keep the old version warm
n_ci->>n_lb: PUT /admin/routes/api HTTP/1.1
n_lb-->>n_ci: HTTP/1.1 200 OK
Note over n_ci,n_lb: Canary: 5% of users first
n_ci->>n_lb: GET /admin/stats?window=3m HTTP/1.1
n_lb-->>n_ci: HTTP/1.1 200 OK
Note over n_ci,n_lb: Rollback by a rule, not by a person
n_ci->>n_lb: PUT /admin/routes/api/green HTTP/1.1
n_lb-->>n_ci: HTTP/1.1 200 OK
Note over n_green,n_flags: Feature flags: deploy is not release
n_green->>n_flags: GET /flags/new-checkout?user=u_42 HTTP/1.1
n_flags-->>n_green: HTTP/1.1 200 OK
Note over n_dev,n_ci: Four numbers to track
n_dev->>n_ci: delivery report, week 46D2
direction: right
n_dev: "Developer" {
shape: person
}
n_users: "Users" {
shape: rectangle
}
n_lb: "Load balancer" {
shape: diamond
}
n_ci: "CI/CD" {
shape: rectangle
style.multiple: true
}
n_registry: "Artifact registry" {
shape: stored_data
}
n_green: "Green servers" {
shape: rectangle
}
n_blue: "Blue servers" {
shape: rectangle
}
n_flags: "Feature flags" {
shape: rectangle
}
n_users -> n_lb
n_dev -> n_ci
n_ci -> n_registry
n_ci -> n_green
n_lb -> n_green
n_lb -> n_blue
n_ci -> n_lb
n_green -> n_flagsPlantUML: request sequence
@startuml
actor "Developer" as n_dev
participant "Blue servers" as n_blue
participant "CI/CD" as n_ci
participant "Load balancer" as n_lb
participant "Green servers" as n_green
participant "Feature flags" as n_flags
note over n_dev, n_blue : Friday deploy, by hand, to six servers
n_dev -> n_blue : $ ssh deploy@192.0.2.12
note over n_dev, n_ci : CI: every commit builds and tests once
n_dev -> n_ci : commit 3f9a1c2 by u_42: checkout, new card form
note over n_lb, n_green : Rolling deploy: one server at a time
n_lb -> n_green : GET /healthz HTTP/1.1
n_green --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_green : Health checks pass, checkout fails
n_lb -> n_green : POST /checkout HTTP/1.1
n_green --> n_lb : HTTP/1.1 500 Internal Server Error
note over n_ci, n_lb : Blue-green: keep the old version warm
n_ci -> n_lb : PUT /admin/routes/api HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_ci, n_lb : Canary: 5% of users first
n_ci -> n_lb : GET /admin/stats?window=3m HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_ci, n_lb : Rollback by a rule, not by a person
n_ci -> n_lb : PUT /admin/routes/api/green HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_green, n_flags : Feature flags: deploy is not release
n_green -> n_flags : GET /flags/new-checkout?user=u_42 HTTP/1.1
n_flags --> n_green : HTTP/1.1 200 OK
note over n_dev, n_ci : Four numbers to track
n_dev -> n_ci : delivery report, week 46
@endumlPlantUML: components
@startuml
left to right direction
actor "Developer" as n_dev
agent "Users" as n_users
node "Load balancer" as n_lb
collections "CI/CD" as n_ci
storage "Artifact registry" as n_registry
component "Green servers" as n_green
component "Blue servers" as n_blue
component "Feature flags" as n_flags
n_users --> n_lb
n_dev --> n_ci
n_ci --> n_registry
n_ci --> n_green
n_lb --> n_green
n_lb --> n_blue
n_ci --> n_lb
n_green --> n_flags
@endumlRecap
Make each deploy small, automatic and easy to undo.
1) CI builds and tests every commit once and publishes one immutable artifact. 2) Rolling needs no extra servers, but rolls back slowly. 3) Blue-green rolls back in a second, at the cost of double capacity. 4) Canary limits a bad release to a few percent, and a rule rolls it back by itself. 5) Feature flags turn code on and off without a deploy. 6) Track deploys per day, lead time, change failure rate and time to restore.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Deploys | 0.1/day | 12/day |
| Failed deploys | 30% | 4% |
| Time to restore | 5 400 s | 90 s |
| Error rate | 40% | 0.3% |
Steps
- Problem Friday deploy, by hand, to six servers
Once a week a developer copies a zip to each of six servers over SSH and restarts the app.
- Fix CI: every commit builds and tests once
Every push now starts a pipeline: build, run the tests, package the result as an image tagged with the commit and put it in a registry.
- Fix Rolling deploy: one server at a time
CI replaces servers one by one.
- Problem Health checks pass, checkout fails
The rollout finished: all six servers run the new version.
- Fix Blue-green: keep the old version warm
Now there are two full sets of servers.
- Fix Canary: 5% of users first
The next release goes to green, but the load balancer sends it only 5% of requests.
- Fix Rollback by a rule, not by a person
The rule is written down in the pipeline: if the canary's error rate is more than twice the stable one for three minutes, set its weight to zero and mark the release bad.
- Fix Feature flags: deploy is not release
The new checkout now ships switched off behind a flag.
- Case Four numbers to track
The team now follows four numbers.
- Recap Recap
Make each deploy small, automatic and easy to undo.
Terms
- CI/CD
- CI builds and tests every commit once and produces one immutable artifact; only green builds deploy. Deploys become small, automatic and easy to undo.
- Blue-green deploy
- Two full sets of servers: the new version goes to the idle set and the load balancer moves all traffic in one step. Rollback takes under a second; the cost is twice the servers.
- Canary deploy
- The new version first gets a small share of traffic, such as 5%, and its error rate is compared with the old one. A bad release reaches few users, and a rule rolls it back by itself.
- Feature flag
- Code ships to production switched off and a flag turns it on for staff, then for a share of users. If it misbehaves, the flag goes off in a second: deploy is not release.