mufassall

Cheat sheet · Security

Secrets management

13 min · Intermediate · mufassall.dtwin.uz/en/l/secrets-management/

← Open the lesson

The problem

A shop backend keeps its database password in a committed .env file shared by five services. It leaks, and the emergency rotation takes checkout down. A secrets manager fixes both: services prove who they are with a workload identity, get their own short-lived database users, secrets are encrypted at rest, keys rotate with overlapping versions, and every read is audited.

What you learn

  • Why a long-lived shared password in a repo or env file is one leak away from a breach
  • Workload identity instead of a static token, and dynamic per-service database users with a lease
  • Envelope encryption at rest and rotation without downtime through two overlapping versions
  • An audit log of every secret read and a break-glass path for when the manager itself is down
Final scene

Diagram as code

The lesson's final design as text: paste it into Jira, GitLab, GitHub, Notion or your docs, where Mermaid renders by itself. There are D2 and PlantUML (Confluence, IntelliJ) versions too.

Mermaid: flowchart

flowchart LR
  n_orders["Orders service"]
  n_reports("Reports job")
  n_db[("Shop DB")]
  n_secrets["Secrets manager"]
  n_issuer>"Identity issuer"]
  n_kms>"KMS"]
  n_partner>"Partner API"]
  n_audit[("Audit log")]
  n_oncall(("On-call"))
  n_safe[("Sealed safe")]
  n_orders --> n_db
  n_reports --> n_db
  n_orders --x n_secrets
  n_reports --x n_secrets
  n_secrets --> n_issuer
  n_secrets --> n_db
  n_secrets --> n_kms
  n_orders --> n_partner
  n_secrets --> n_partner
  n_secrets --> n_audit
  n_oncall --> n_safe
  n_oncall -.->|"retry"| n_db
  classDef down fill:#fde2e2,stroke:#c62828,color:#4a0d0d,stroke-dasharray:4 3
  class n_secrets down

Mermaid: request sequence

sequenceDiagram
  participant n_repo as Git repo
  participant n_orders as Orders service
  participant n_intruder as Intruder
  participant n_db as Shop DB
  participant n_secrets as Secrets manager
  participant n_kms as KMS
  participant n_partner as Partner API
  participant n_audit as Audit log
  actor n_oncall as On-call
  participant n_safe as Sealed safe
  Note over n_repo,n_orders: The password lives in a file in the repo
  n_repo->>n_orders: $ cat services/orders/.env
  Note over n_intruder,n_db: A backup of the repo leaks, someone logs in
  n_intruder->>n_db: 02:14:07 LOG connection authorized: user=app database=shop
  Note over n_orders,n_db: Emergency rotation takes checkout down
  n_orders->>n_db: 02:31:00 admin ALTER ROLE app PASSWORD '***'
  Note over n_orders,n_secrets: Secrets move to a secrets manager
  n_orders->>n_secrets: GET /v1/secret/shop/db HTTP/1.1
  n_secrets-->>n_orders: HTTP/1.1 200 OK
  Note over n_orders,n_secrets: The service proves who it is, no stored token
  n_orders->>n_secrets: POST /v1/auth/workload/login HTTP/1.1
  n_secrets-->>n_orders: HTTP/1.1 200 OK
  Note over n_orders,n_secrets: Each service gets its own database user with a lease
  n_orders->>n_secrets: GET /v1/database/creds/orders-rw HTTP/1.1
  n_secrets-->>n_orders: HTTP/1.1 200 OK
  Note over n_secrets,n_db: What the manager does in the database
  n_secrets->>n_db: CREATE ROLE #34;v-orders-h7Kq#34; LOGIN PASSWORD '***'
  n_db-->>n_secrets: CREATE ROLE
  Note over n_secrets,n_kms: Secrets at rest are encrypted, the root key stays in KMS
  n_secrets->>n_kms: POST /v1/keys/secrets-root/decrypt HTTP/1.1
  n_kms-->>n_secrets: HTTP/1.1 200 OK
  Note over n_secrets,n_partner: Rotation without downtime: two versions overlap
  n_secrets->>n_partner: 03:00:00 rotate partner/api-key: created version 8
  Note over n_secrets,n_audit: Every read is logged, a wrong read is visible
  n_secrets->>n_audit: audit.log
  Note over n_oncall,n_safe: When the manager is down: break-glass
  n_oncall->>n_safe: 04:12 BREAK-GLASS shop-db opened by oncall_1

D2

direction: right

n_orders: "Orders service" {
  shape: rectangle
}
n_reports: "Reports job" {
  shape: rectangle
  style.multiple: true
}
n_db: "Shop DB" {
  shape: cylinder
}
n_secrets: "Secrets manager" {
  shape: rectangle
  style.fill: "#fde2e2"
  style.stroke: "#c62828"
  style.stroke-dash: 3
}
n_issuer: "Identity issuer" {
  shape: cloud
}
n_kms: "KMS" {
  shape: cloud
}
n_partner: "Partner API" {
  shape: cloud
}
n_audit: "Audit log" {
  shape: stored_data
}
n_oncall: "On-call" {
  shape: person
}
n_safe: "Sealed safe" {
  shape: stored_data
}

n_orders -> n_db
n_reports -> n_db
n_orders -> n_secrets {
  target-arrowhead.shape: cross
}
n_reports -> n_secrets {
  target-arrowhead.shape: cross
}
n_secrets -> n_issuer
n_secrets -> n_db
n_secrets -> n_kms
n_orders -> n_partner
n_secrets -> n_partner
n_secrets -> n_audit
n_oncall -> n_safe
n_oncall -> n_db: "retry" {
  style.stroke-dash: 3
}

PlantUML: request sequence

@startuml
database "Git repo" as n_repo
participant "Orders service" as n_orders
boundary "Intruder" as n_intruder
database "Shop DB" as n_db
participant "Secrets manager" as n_secrets
boundary "KMS" as n_kms
boundary "Partner API" as n_partner
database "Audit log" as n_audit
actor "On-call" as n_oncall
database "Sealed safe" as n_safe
note over n_repo, n_orders : The password lives in a file in the repo
n_repo -> n_orders : $ cat services/orders/.env
note over n_intruder, n_db : A backup of the repo leaks, someone logs in
n_intruder -> n_db : 02:14:07 LOG connection authorized: user=app database=shop
note over n_orders, n_db : Emergency rotation takes checkout down
n_orders -> n_db : 02:31:00 admin ALTER ROLE app PASSWORD '***'
note over n_orders, n_secrets : Secrets move to a secrets manager
n_orders -> n_secrets : GET /v1/secret/shop/db HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_secrets : The service proves who it is, no stored token
n_orders -> n_secrets : POST /v1/auth/workload/login HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_orders, n_secrets : Each service gets its own database user with a lease
n_orders -> n_secrets : GET /v1/database/creds/orders-rw HTTP/1.1
n_secrets --> n_orders : HTTP/1.1 200 OK
note over n_secrets, n_db : What the manager does in the database
n_secrets -> n_db : CREATE ROLE "v-orders-h7Kq" LOGIN PASSWORD '***'
n_db --> n_secrets : CREATE ROLE
note over n_secrets, n_kms : Secrets at rest are encrypted, the root key stays in KMS
n_secrets -> n_kms : POST /v1/keys/secrets-root/decrypt HTTP/1.1
n_kms --> n_secrets : HTTP/1.1 200 OK
note over n_secrets, n_partner : Rotation without downtime: two versions overlap
n_secrets -> n_partner : 03:00:00 rotate partner/api-key: created version 8
note over n_secrets, n_audit : Every read is logged, a wrong read is visible
n_secrets -> n_audit : audit.log
note over n_oncall, n_safe : When the manager is down: break-glass
n_oncall -> n_safe : 04:12 BREAK-GLASS shop-db opened by oncall_1
@enduml

PlantUML: components

@startuml
left to right direction
component "Orders service" as n_orders
collections "Reports job" as n_reports
database "Shop DB" as n_db
component "Secrets manager" as n_secrets #fde2e2;line:c62828;line.dashed
cloud "Identity issuer" as n_issuer
cloud "KMS" as n_kms
cloud "Partner API" as n_partner
storage "Audit log" as n_audit
actor "On-call" as n_oncall
storage "Sealed safe" as n_safe
n_orders --> n_db
n_reports --> n_db
n_orders -[#c62828]-> n_secrets
n_reports -[#c62828]-> n_secrets
n_secrets --> n_issuer
n_secrets --> n_db
n_secrets --> n_kms
n_orders --> n_partner
n_secrets --> n_partner
n_secrets --> n_audit
n_oncall --> n_safe
n_oncall ..> n_db : retry
@enduml

Recap

Six rules for secrets.

1) No secrets in repos, images or env files; anything that was in git is leaked. 2) Services prove who they are with a workload identity, not a stored token. 3) Prefer dynamic, per-service credentials with a lease and least privilege. 4) Encrypt secrets at rest with envelope encryption, root key in a KMS. 5) Rotate with two overlapping versions and revoke only after the old one is unused. 6) Audit every read, alert on denials, and keep a tested two-person break-glass path.

Numbers: at the start and at the end

MetricStartEnd
Static secret copies91
Services per leak51
Rotation downtime0 s0 s
Reads audited0%100%

Steps

  1. Case The password lives in a file in the repo

    The orders service reads its database login from a .env file.

  2. Problem A backup of the repo leaks, someone logs in

    A laptop backup with the repo was public for three days.

  3. Problem Emergency rotation takes checkout down

    The admin changes the password at 02:31.

  4. Fix Secrets move to a secrets manager

    The password leaves the repo, and git history is cleaned.

  5. Fix The service proves who it is, no stored token

    The platform that runs the pods gives each one a signed, short-lived token that says "I am the orders service in the shop namespace".

  6. Fix Each service gets its own database user with a lease

    Instead of reading a stored password, the orders service asks for credentials.

  7. Case What the manager does in the database

    The manager holds one admin login to the database, used only to create and drop users.

  8. Fix Secrets at rest are encrypted, the root key stays in KMS

    The manager stores secrets in its own storage, and that storage gets backed up.

  9. Fix Rotation without downtime: two versions overlap

    Some secrets cannot be dynamic: the partner's API key is issued by the partner.

  10. Fix Every read is logged, a wrong read is visible

    Every request to the manager writes an audit record: which workload, which path, the result and from where.

  11. Case When the manager is down: break-glass

    At 04:12 the secrets manager is down and services cannot get credentials.

  12. Recap Recap

    Six rules for secrets.

Terms

Secrets manager
A service that stores secrets, checks who is asking and returns only what that caller may read, auditing every read.
Dynamic credentials
A database user or key created per service on request and deleted when its lease expires.
Workload identity
A signed, short-lived token the platform gives a service so it can prove who it is without a stored secret.