Cheat sheet · Security
Secrets management
The problem
A shop backend keeps its database password in a committed .env file shared by five services. It leaks, and the emergency rotation takes checkout down. A secrets manager fixes both: services prove who they are with a workload identity, get their own short-lived database users, secrets are encrypted at rest, keys rotate with overlapping versions, and every read is audited.
What you learn
- Why a long-lived shared password in a repo or env file is one leak away from a breach
- Workload identity instead of a static token, and dynamic per-service database users with a lease
- Envelope encryption at rest and rotation without downtime through two overlapping versions
- An audit log of every secret read and a break-glass path for when the manager itself is down
Diagram as code
The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.
Mermaid: flowchart
flowchart LR
n_orders["Orders service"]
n_reports("Reports job")
n_db[("Shop DB")]
n_secrets["Secrets manager"]
n_issuer>"Identity issuer"]
n_kms>"KMS"]
n_partner>"Partner API"]
n_audit[("Audit log")]
n_oncall(("On-call"))
n_safe[("Sealed safe")]
n_orders --> n_db
n_reports --> n_db
n_orders --x n_secrets
n_reports --x n_secrets
n_secrets --> n_issuer
n_secrets --> n_db
n_secrets --> n_kms
n_orders --> n_partner
n_secrets --> n_partner
n_secrets --> n_audit
n_oncall --> n_safe
n_oncall -.->|"retry"| n_db
classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
class n_secrets downMermaid: request sequence
sequenceDiagram
participant n_repo as Git repo
participant n_orders as Orders service
participant n_intruder as Intruder
participant n_db as Shop DB
participant n_secrets as Secrets manager
participant n_kms as KMS
participant n_partner as Partner API
participant n_audit as Audit log
actor n_oncall as On-call
participant n_safe as Sealed safe
Note over n_repo,n_orders: The password lives in a file in the repo
n_repo->>n_orders: $ cat services/orders/.env
Note over n_intruder,n_db: A backup of the repo leaks, someone logs in
n_intruder->>n_db: 02:14:07 LOG connection authorized: user=app database=shop
Note over n_orders,n_db: Emergency rotation takes checkout down
n_orders->>n_db: 02:31:00 admin ALTER ROLE app PASSWORD '***'
Note over n_orders,n_secrets: Secrets move to a secrets manager
n_orders->>n_secrets: GET /v1/secret/shop/db HTTP/1.1
n_secrets-->>n_orders: HTTP/1.1 200 OK
Note over n_orders,n_secrets: The service proves who it is, no stored token
n_orders->>n_secrets: POST /v1/auth/workload/login HTTP/1.1
n_secrets-->>n_orders: HTTP/1.1 200 OK
Note over n_orders,n_secrets: Each service gets its own database user with a lease
n_orders->>n_secrets: GET /v1/database/creds/orders-rw HTTP/1.1
n_secrets-->>n_orders: HTTP/1.1 200 OK
Note over n_secrets,n_db: What the manager does in the database
n_secrets->>n_db: CREATE ROLE #34;v-orders-h7Kq#34; LOGIN PASSWORD '***'
n_db-->>n_secrets: CREATE ROLE
Note over n_secrets,n_kms: Secrets at rest are encrypted, the root key stays in KMS
n_secrets->>n_kms: POST /v1/keys/secrets-root/decrypt HTTP/1.1
n_kms-->>n_secrets: HTTP/1.1 200 OK
Note over n_secrets,n_partner: Rotation without downtime: two versions overlap
n_secrets->>n_partner: 03:00:00 rotate partner/api-key: created version 8
Note over n_secrets,n_audit: Every read is logged, a wrong read is visible
n_secrets->>n_audit: audit.log
Note over n_oncall,n_safe: When the manager is down: break-glass
n_oncall->>n_safe: 04:12 BREAK-GLASS shop-db opened by oncall_1D2
direction: right
n_orders: "Orders service" {
shape: rectangle
}
n_reports: "Reports job" {
shape: rectangle
style.multiple: true
}
n_db: "Shop DB" {
shape: cylinder
}
n_secrets: "Secrets manager" {
shape: rectangle
style.fill: "#fde2e2"
style.stroke: "#c62828"
style.stroke-dash: 3
}
n_issuer: "Identity issuer" {
shape: cloud
}
n_kms: "KMS" {
shape: cloud
}
n_partner: "Partner API" {
shape: cloud
}
n_audit: "Audit log" {
shape: stored_data
}
n_oncall: "On-call" {
shape: person
}
n_safe: "Sealed safe" {
shape: stored_data
}
n_orders -> n_db
n_reports -> n_db
n_orders -> n_secrets {
target-arrowhead.shape: cross
}
n_reports -> n_secrets {
target-arrowhead.shape: cross
}
n_secrets -> n_issuer
n_secrets -> n_db
n_secrets -> n_kms
n_orders -> n_partner
n_secrets -> n_partner
n_secrets -> n_audit
n_oncall -> n_safe
n_oncall -> n_db: "retry" {
style.stroke-dash: 3
}PlantUML: request sequence
@startuml
database "Git repo" as n_repo
participant "Orders service" as n_orders
boundary "Intruder" as n_intruder
database "Shop DB" as n_db
participant "Secrets manager" as n_secrets
boundary "KMS" as n_kms
boundary "Partner API" as n_partner
database "Audit log" as n_audit
actor "On-call" as n_oncall
database "Sealed safe" as n_safe
note over n_repo, n_orders : The password lives in a file in the repo
n_repo -> n_orders : $ cat services/orders/.env
note over n_intruder, n_db : A backup of the repo leaks, someone logs in
n_intruder -> n_db : 02:14:07 LOG connection authorized: user=app database=shop
note over n_orders, n_db : Emergency rotation takes checkout down
n_orders -> n_db : 02:31:00 admin ALTER ROLE app PASSWORD '***'
note over n_orders, n_secrets : Secrets move to a secrets manager
n_orders -> n_secrets : GET /v1/secret/shop/db HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_secrets : The service proves who it is, no stored token
n_orders -> n_secrets : POST /v1/auth/workload/login HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_secrets : Each service gets its own database user with a lease
n_orders -> n_secrets : GET /v1/database/creds/orders-rw HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_secrets, n_db : What the manager does in the database
n_secrets -> n_db : CREATE ROLE "v-orders-h7Kq" LOGIN PASSWORD '***'
n_db --> n_secrets : CREATE ROLE
note over n_secrets, n_kms : Secrets at rest are encrypted, the root key stays in KMS
n_secrets -> n_kms : POST /v1/keys/secrets-root/decrypt HTTP/1.1
n_kms --> n_secrets : HTTP/1.1 200 OK
note over n_secrets, n_partner : Rotation without downtime: two versions overlap
n_secrets -> n_partner : 03:00:00 rotate partner/api-key: created version 8
note over n_secrets, n_audit : Every read is logged, a wrong read is visible
n_secrets -> n_audit : audit.log
note over n_oncall, n_safe : When the manager is down: break-glass
n_oncall -> n_safe : 04:12 BREAK-GLASS shop-db opened by oncall_1
@endumlPlantUML: components
@startuml
left to right direction
component "Orders service" as n_orders
collections "Reports job" as n_reports
database "Shop DB" as n_db
component "Secrets manager" as n_secrets #fde2e2;line:c62828;line.dashed
cloud "Identity issuer" as n_issuer
cloud "KMS" as n_kms
cloud "Partner API" as n_partner
storage "Audit log" as n_audit
actor "On-call" as n_oncall
storage "Sealed safe" as n_safe
n_orders --> n_db
n_reports --> n_db
n_orders -[#c62828]-> n_secrets
n_reports -[#c62828]-> n_secrets
n_secrets --> n_issuer
n_secrets --> n_db
n_secrets --> n_kms
n_orders --> n_partner
n_secrets --> n_partner
n_secrets --> n_audit
n_oncall --> n_safe
n_oncall ..> n_db : retry
@endumlRecap
Six rules for secrets.
1) No secrets in repos, images or env files; anything that was in git is leaked. 2) Services prove who they are with a workload identity, not a stored token. 3) Prefer dynamic, per-service credentials with a lease and least privilege. 4) Encrypt secrets at rest with envelope encryption, root key in a KMS. 5) Rotate with two overlapping versions and revoke only after the old one is unused. 6) Audit every read, alert on denials, and keep a tested two-person break-glass path.
Numbers: at the start and at the end
| Metric | Start | End |
|---|---|---|
| Static secret copies | 9 | 1 |
| Services per leak | 5 | 1 |
| Rotation downtime | 0 s | 0 s |
| Reads audited | 0% | 100% |
Steps
- Case The password lives in a file in the repo
The orders service reads its database login from a
.envfile. - Problem A backup of the repo leaks, someone logs in
A laptop backup with the repo was public for three days.
- Problem Emergency rotation takes checkout down
The admin changes the password at 02:31.
- Fix Secrets move to a secrets manager
The password leaves the repo, and git history is cleaned.
- Fix The service proves who it is, no stored token
The platform that runs the pods gives each one a signed, short-lived token that says "I am the orders service in the shop namespace".
- Fix Each service gets its own database user with a lease
Instead of reading a stored password, the orders service asks for credentials.
- Case What the manager does in the database
The manager holds one admin login to the database, used only to create and drop users.
- Fix Secrets at rest are encrypted, the root key stays in KMS
The manager stores secrets in its own storage, and that storage gets backed up.
- Fix Rotation without downtime: two versions overlap
Some secrets cannot be dynamic: the partner's API key is issued by the partner.
- Fix Every read is logged, a wrong read is visible
Every request to the manager writes an audit record: which workload, which path, the result and from where.
- Case When the manager is down: break-glass
At 04:12 the secrets manager is down and services cannot get credentials.
- Recap Recap
Six rules for secrets.
Terms
- Secrets manager
- A service that stores secrets, checks who is asking and returns only what that caller may read, auditing every read.
- Dynamic credentials
- A database user or key created per service on request and deleted when its lease expires.
- Workload identity
- A signed, short-lived token the platform gives a service so it can prove who it is without a stored secret.