mufassall

Cheat sheet · Shipping

CI/CD: from commit to production

14 min · Intermediate · mufassall.dtwin.uz/en/l/cicd/

← Open the lesson

The problem

A team deploys once a week, by hand, and every third deploy breaks something. Step by step the path from commit to production becomes automatic, small and reversible, and the team ends up deploying a dozen times a day.

What you learn

  • CI builds and tests every commit once and produces one immutable artifact
  • Rolling, blue-green and canary deploys: what each costs and how fast each rolls back
  • Automatic rollback on error rate, and feature flags that separate deploy from release
  • Four delivery numbers: deploys per day, lead time, change failure rate, time to restore
Final scene
45%

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_dev(("Developer"))
  n_users(["Users"])
  n_lb{"Load balancer"}
  n_ci("CI/CD")
  n_registry[("Artifact registry")]
  n_green["Green servers"]
  n_blue["Blue servers"]
  n_flags["Feature flags"]
  n_users --> n_lb
  n_dev --> n_ci
  n_ci --> n_registry
  n_ci --> n_green
  n_lb --> n_green
  n_lb --> n_blue
  n_ci --> n_lb
  n_green --> n_flags

Mermaid: request sequence

sequenceDiagram
  actor n_dev as Developer
  participant n_blue as Blue servers
  participant n_ci as CI/CD
  participant n_lb as Load balancer
  participant n_green as Green servers
  participant n_flags as Feature flags
  Note over n_dev,n_blue: Friday deploy, by hand, to six servers
  n_dev->>n_blue: $ ssh deploy@192.0.2.12
  Note over n_dev,n_ci: CI: every commit builds and tests once
  n_dev->>n_ci: commit 3f9a1c2 by u_42: checkout, new card form
  Note over n_lb,n_green: Rolling deploy: one server at a time
  n_lb->>n_green: GET /healthz HTTP/1.1
  n_green-->>n_lb: HTTP/1.1 200 OK
  Note over n_lb,n_green: Health checks pass, checkout fails
  n_lb->>n_green: POST /checkout HTTP/1.1
  n_green-->>n_lb: HTTP/1.1 500 Internal Server Error
  Note over n_ci,n_lb: Blue-green: keep the old version warm
  n_ci->>n_lb: PUT /admin/routes/api HTTP/1.1
  n_lb-->>n_ci: HTTP/1.1 200 OK
  Note over n_ci,n_lb: Canary: 5% of users first
  n_ci->>n_lb: GET /admin/stats?window=3m HTTP/1.1
  n_lb-->>n_ci: HTTP/1.1 200 OK
  Note over n_ci,n_lb: Rollback by a rule, not by a person
  n_ci->>n_lb: PUT /admin/routes/api/green HTTP/1.1
  n_lb-->>n_ci: HTTP/1.1 200 OK
  Note over n_green,n_flags: Feature flags: deploy is not release
  n_green->>n_flags: GET /flags/new-checkout?user=u_42 HTTP/1.1
  n_flags-->>n_green: HTTP/1.1 200 OK
  Note over n_dev,n_ci: Four numbers to track
  n_dev->>n_ci: delivery report, week 46

D2

direction: right

n_dev: "Developer" {
  shape: person
}
n_users: "Users" {
  shape: rectangle
}
n_lb: "Load balancer" {
  shape: diamond
}
n_ci: "CI/CD" {
  shape: rectangle
  style.multiple: true
}
n_registry: "Artifact registry" {
  shape: stored_data
}
n_green: "Green servers" {
  shape: rectangle
}
n_blue: "Blue servers" {
  shape: rectangle
}
n_flags: "Feature flags" {
  shape: rectangle
}

n_users -> n_lb
n_dev -> n_ci
n_ci -> n_registry
n_ci -> n_green
n_lb -> n_green
n_lb -> n_blue
n_ci -> n_lb
n_green -> n_flags

PlantUML: request sequence

@startuml
actor "Developer" as n_dev
participant "Blue servers" as n_blue
participant "CI/CD" as n_ci
participant "Load balancer" as n_lb
participant "Green servers" as n_green
participant "Feature flags" as n_flags
note over n_dev, n_blue : Friday deploy, by hand, to six servers
n_dev -> n_blue : $ ssh deploy@192.0.2.12
note over n_dev, n_ci : CI: every commit builds and tests once
n_dev -> n_ci : commit 3f9a1c2 by u_42: checkout, new card form
note over n_lb, n_green : Rolling deploy: one server at a time
n_lb -> n_green : GET /healthz HTTP/1.1
n_green --> n_lb : HTTP/1.1 200 OK
note over n_lb, n_green : Health checks pass, checkout fails
n_lb -> n_green : POST /checkout HTTP/1.1
n_green --> n_lb : HTTP/1.1 500 Internal Server Error
note over n_ci, n_lb : Blue-green: keep the old version warm
n_ci -> n_lb : PUT /admin/routes/api HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_ci, n_lb : Canary: 5% of users first
n_ci -> n_lb : GET /admin/stats?window=3m HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_ci, n_lb : Rollback by a rule, not by a person
n_ci -> n_lb : PUT /admin/routes/api/green HTTP/1.1
n_lb --> n_ci : HTTP/1.1 200 OK
note over n_green, n_flags : Feature flags: deploy is not release
n_green -> n_flags : GET /flags/new-checkout?user=u_42 HTTP/1.1
n_flags --> n_green : HTTP/1.1 200 OK
note over n_dev, n_ci : Four numbers to track
n_dev -> n_ci : delivery report, week 46
@enduml

PlantUML: components

@startuml
left to right direction
actor "Developer" as n_dev
agent "Users" as n_users
node "Load balancer" as n_lb
collections "CI/CD" as n_ci
storage "Artifact registry" as n_registry
component "Green servers" as n_green
component "Blue servers" as n_blue
component "Feature flags" as n_flags
n_users --> n_lb
n_dev --> n_ci
n_ci --> n_registry
n_ci --> n_green
n_lb --> n_green
n_lb --> n_blue
n_ci --> n_lb
n_green --> n_flags
@enduml

Recap

Make each deploy small, automatic and easy to undo.

1) CI builds and tests every commit once and publishes one immutable artifact. 2) Rolling needs no extra servers, but rolls back slowly. 3) Blue-green rolls back in a second, at the cost of double capacity. 4) Canary limits a bad release to a few percent, and a rule rolls it back by itself. 5) Feature flags turn code on and off without a deploy. 6) Track deploys per day, lead time, change failure rate and time to restore.

Numbers: at the start and at the end

MetricStartEnd
Deploys0.1/day12/day
Failed deploys30%4%
Time to restore5 400 s90 s
Error rate40%0.3%

Steps

  1. Problem Friday deploy, by hand, to six servers

    Once a week a developer copies a zip to each of six servers over SSH and restarts the app.

  2. Fix CI: every commit builds and tests once

    Every push now starts a pipeline: build, run the tests, package the result as an image tagged with the commit and put it in a registry.

  3. Fix Rolling deploy: one server at a time

    CI replaces servers one by one.

  4. Problem Health checks pass, checkout fails

    The rollout finished: all six servers run the new version.

  5. Fix Blue-green: keep the old version warm

    Now there are two full sets of servers.

  6. Fix Canary: 5% of users first

    The next release goes to green, but the load balancer sends it only 5% of requests.

  7. Fix Rollback by a rule, not by a person

    The rule is written down in the pipeline: if the canary's error rate is more than twice the stable one for three minutes, set its weight to zero and mark the release bad.

  8. Fix Feature flags: deploy is not release

    The new checkout now ships switched off behind a flag.

  9. Case Four numbers to track

    The team now follows four numbers.

  10. Recap Recap

    Make each deploy small, automatic and easy to undo.

Terms

CI/CD
CI builds and tests every commit once and produces one immutable artifact; only green builds deploy. Deploys become small, automatic and easy to undo.
Blue-green deploy
Two full sets of servers: the new version goes to the idle set and the load balancer moves all traffic in one step. Rollback takes under a second; the cost is twice the servers.
Canary deploy
The new version first gets a small share of traffic, such as 5%, and its error rate is compared with the old one. A bad release reaches few users, and a rule rolls it back by itself.
Feature flag
Code ships to production switched off and a flag turns it on for staff, then for a share of users. If it misbehaves, the flag goes off in a second: deploy is not release.