Secrets Management: HashiCorp Vault, KMS & HSM

Chapter 34: Secrets Management: HashiCorp Vault, KMS & HSM

Learning Objectives

• Understand the secrets management problem and anti-patterns
• Configure and use HashiCorp Vault in production
• Use cloud KMS (AWS KMS, GCP KMS) for envelope encryption
• Integrate secrets into Kubernetes with External Secrets Operator

---

What is this? (Beginner On-Ramp)

Secrets management is how we safely store sensitive data like database passwords and API keys. Instead of writing passwords in code where anyone can see them, applications ask a secure digital vault for the password exactly when they need it.

34.1 The Secrets Problem

    Secrets Anti-Patterns (NEVER DO THESE)
    ────────────────────────────────────────

    ✗ Hardcoded in source code:
      password = "super_secret_123"   ← checked into git forever

    ✗ In environment variables (in plaintext):
      docker run -e DB_PASSWORD=secret myapp
      → Visible in: docker inspect, ps aux, /proc/PID/environ

    ✗ In Kubernetes Secrets without encryption at rest:
      Kubernetes Secrets are base64-encoded, NOT encrypted
      → Stored in etcd in plaintext by default!

    ✗ Shared passwords:
      All DBAs use "dba_password_2020" → impossible to audit

    ✗ Long-lived credentials:
      API key never rotated → if leaked, exposed forever

    Best Practices:
    ────────────────
    ✓ No secrets in code or config files
    ✓ Short-lived credentials (rotate or expire often)
    ✓ Audit every access to secrets
    ✓ Least privilege: service only gets its own secrets
    ✓ Secrets management system (Vault, KMS)

---

34.2 HashiCorp Vault Architecture

    Vault Architecture
    ───────────────────

    ┌───────────────────────────────────────────────────────┐
    │                    VAULT SERVER                        │
    │                                                        │
    │  ┌─────────────┐   ┌────────────┐  ┌──────────────┐  │
    │  │  Auth       │   │  Secrets   │  │  Audit       │  │
    │  │  Methods    │   │  Engines   │  │  Backend     │  │
    │  │             │   │            │  │              │  │
    │  │  AppRole    │   │  KV v2     │  │  File log    │  │
    │  │  Kubernetes │   │  Database  │  │  Syslog      │  │
    │  │  AWS IAM    │   │  PKI       │  │              │  │
    │  │  JWT/OIDC   │   │  AWS/GCP   │  └──────────────┘  │
    │  └─────────────┘   └────────────┘                     │
    │                                                        │
    │  ┌─────────────────────────────────────────────────┐  │
    │  │  Policy Engine (who can access what)             │  │
    │  └─────────────────────────────────────────────────┘  │
    └───────────────────────────────────────────────────────┘
             │ Encrypted
             ▼
    ┌─────────────────────┐
    │  Storage Backend    │
    │  Consul / etcd /    │
    │  S3 / PostgreSQL    │
    └─────────────────────┘

    Vault Seal/Unseal:
    ──────────────────
    Vault encrypts all data with a master key
    Master key is split (Shamir) into N key shares
    Need M of N shares to unseal (M-of-N threshold)
    OR use auto-unseal with cloud KMS (AWS KMS seals the master key)

---

34.3 Vault Setup and Basic Usage

# ── Initialize Vault ──────────────────────────────────────
vault operator init -key-shares=5 -key-threshold=3
# Generates: 5 unseal key shares, 1 initial root token
# Save unseal keys in secure separate locations!

# Unseal (need 3 of 5 keys)
vault operator unseal <key1>
vault operator unseal <key2>
vault operator unseal <key3>

vault status
# Sealed: false

# ── KV (Key-Value) Secrets Engine ────────────────────────
# Enable KV v2
vault secrets enable -path=secret kv-v2

# Store a secret
vault kv put secret/production/database \
    host="db.internal" \
    port="5432" \
    username="app_user" \
    password="$(openssl rand -base64 32)"

# Read a secret
vault kv get secret/production/database
vault kv get -field=password secret/production/database

# List secrets
vault kv list secret/production/

# Update (creates new version)
vault kv put secret/production/database password="new_password"

# Read specific version
vault kv get -version=2 secret/production/database

# ── Dynamic Database Credentials ──────────────────────────
# Vault generates short-lived DB credentials on demand!

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/production \
    plugin_name=postgresql-database-plugin \
    allowed_roles="app-role" \
    connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/appdb" \
    username="vault_admin" \
    password="vault_admin_password"

# Create a role (TTL: credential lifetime)
vault write database/roles/app-role \
    db_name=production \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT,INSERT,UPDATE,DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

# App requests credentials (gets fresh user each time!)
vault read database/creds/app-role
# Key                Value
# lease_duration     1h
# username           v-app-role-AbCdEfGh
# password           A1b2C3d4E5f6G7h8
# → Credentials auto-expire after 1 hour!

---

34.4 Vault Policies

/etc/vault/policies/checkout-service.hcl

# Allow checkout-service to read its own secrets
path "secret/data/production/checkout/*" {
  capabilities = ["read"]
}

# Allow requesting database credentials
path "database/creds/checkout-db-role" {
  capabilities = ["read"]
}

# Allow generating TLS certificates
path "pki/issue/checkout-service" {
  capabilities = ["create", "update"]
}

# Deny everything else (implicit)

# Apply policy
vault policy write checkout-service /etc/vault/policies/checkout-service.hcl

# Create AppRole for checkout-service
vault auth enable approle
vault write auth/approle/role/checkout-service \
    token_policies="checkout-service" \
    token_ttl=1h \
    token_max_ttl=4h

# Get credentials for the app
ROLE_ID=$(vault read -field=role_id auth/approle/role/checkout-service/role-id)
SECRET_ID=$(vault write -field=secret_id -f auth/approle/role/checkout-service/secret-id)

# App authenticates:
TOKEN=$(vault write -field=token auth/approle/login \
    role_id=$ROLE_ID secret_id=$SECRET_ID)

---

34.5 Kubernetes + Vault Integration

# Vault Agent Injector (sidecar pattern)
# Automatically injects secrets into pods as files

# 1. Enable Kubernetes auth
# vault auth enable kubernetes
# vault write auth/kubernetes/config \
#   kubernetes_host="https://kubernetes.default.svc:443" \
#   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# 2. Annotate pods to get secrets injected
apiVersion: v1
kind: Pod
metadata:
  name: checkout-service
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "checkout-service"
    vault.hashicorp.com/agent-inject-secret-database: "secret/data/production/database"
    vault.hashicorp.com/agent-inject-template-database: |
      {{- with secret "secret/data/production/database" -}}
      DB_HOST={{ .Data.data.host }}
      DB_PASSWORD={{ .Data.data.password }}
      {{- end }}
spec:
  serviceAccountName: checkout-service
  containers:
  - name: app
    image: checkout:latest
    # Secrets available at: /vault/secrets/database
    command: ["sh", "-c", "source /vault/secrets/database && ./app"]

# External Secrets Operator (alternative approach)
# Pulls secrets from Vault/AWS SM and creates k8s Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-secret
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: checkout-db-credentials    # Creates this k8s Secret
  data:
  - secretKey: DB_PASSWORD
    remoteRef:
      key: production/database
      property: password

---

34.6 Cloud KMS & Envelope Encryption

    Envelope Encryption
    ────────────────────

    Problem: Encrypting 1GB file with KMS is slow and expensive

    Solution: Envelope Encryption
    1. Generate local DEK (Data Encryption Key) — random, ephemeral
    2. Encrypt your data with DEK (fast, local)
    3. Encrypt DEK with KMS CMK (only small key sent to KMS)
    4. Store: encrypted data + encrypted DEK together

    Decrypt:
    1. Send encrypted DEK to KMS → get DEK back
    2. Decrypt data with DEK locally

    Result: KMS only ever sees small DEKs, not bulk data

# AWS KMS
# Create CMK
aws kms create-key --description "App encryption key"

# Encrypt data
aws kms encrypt \
    --key-id arn:aws:kms:us-east-1:123456789:key/mrk-xxx \
    --plaintext fileb://secret.txt \
    --output text --query CiphertextBlob | base64 --decode > encrypted.bin

# Decrypt
aws kms decrypt \
    --ciphertext-blob fileb://encrypted.bin \
    --output text --query Plaintext | base64 --decode

# Enable automatic key rotation (annually)
aws kms enable-key-rotation --key-id arn:aws:kms:...:key/mrk-xxx

# GCP KMS
gcloud kms keys create my-key \
    --keyring my-keyring \
    --location global \
    --purpose encryption \
    --rotation-period 90d

gcloud kms encrypt \
    --plaintext-file=secret.txt \
    --ciphertext-file=encrypted.bin \
    --key=my-key --keyring=my-keyring --location=global

---

34.7 Interview Questions

Q1: What is the difference between Vault’s KV engine and Dynamic secrets?

Answer: KV (Key-Value) is static secret storage — you store a password and retrieve it when needed. It’s encrypted at rest and access is audited, but the secret itself is long-lived. Dynamic secrets are generated on-demand and short-lived. For databases, Vault creates a new user with a unique password when an app requests credentials, with a TTL (e.g., 1 hour). After the TTL expires, Vault automatically deletes the user. Benefits: (1) No shared passwords — each service instance gets its own unique credential. (2) Automatic expiry — even if leaked, useless in 1 hour. (3) Complete audit trail — every credential request is logged with the requester identity.

Q2: What is envelope encryption and why is it used with KMS?

Answer: Envelope encryption solves the problem of encrypting large amounts of data efficiently with KMS. You generate a local Data Encryption Key (DEK), encrypt your data locally with it (fast), then send only the small DEK to KMS to be encrypted with the Customer Master Key (CMK). Store the encrypted data together with the encrypted DEK. To decrypt: send the encrypted DEK to KMS → get the DEK back → decrypt data locally. This is efficient because: KMS only processes small keys (not bulk data), encryption/decryption is fast (done locally with AES), and the DEK is protected by KMS (can’t be decrypted without calling KMS, which is audited).

---

34.8 Summary

Tool/Concept	Purpose
Vault KV	Static encrypted secret storage
Vault Dynamic	Short-lived generated credentials
Vault PKI	Internal CA for TLS certificates
AppRole	Machine authentication to Vault
Kubernetes auth	Pod SA-based Vault authentication
Envelope encryption	Efficient KMS-protected data encryption
External Secrets	Sync Vault/KMS secrets to k8s Secrets

---

Next Chapter: Chapter 35: Container Security

Prerequisites

Basic security principles.

---

Advantages & Disadvantages

Advantages: Prevents leaked credentials in GitHub, dynamic secrets limit exposure time. Disadvantages: Vault becomes a massive single point of failure (if Vault is down, everything is down).

---

Common Mistakes

• Committing secrets (API keys, passwords) to version control.
• Passing secrets as plain text environment variables in Docker (visible via docker inspect).
• Using long-lived static credentials instead of dynamic, short-lived ones.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
App cannot access database	Vault token expired or missing policy	Check app logs for 403 Forbidden from Vault	Renew token or attach correct Vault policy
Secret leaked to GitHub	Accidental commit	git log -p	Revoke secret immediately, rotate it, and rewrite git history (or use BFG)

---

Hands-on Lab

Objective: Interact with a secrets manager (Vault).

# 1. Start a dev Vault server (requires vault installed)
# vault server -dev

# 2. Write a secret
# vault kv put secret/myapp/config db_password=supersecret

# 3. Read a secret
# vault kv get secret/myapp/config

---

Exercises

1. Integrate a Python script with HashiCorp Vault to retrieve database credentials at startup instead of reading them from a file.
2. Configure a Vault dynamic secrets engine to generate short-lived PostgreSQL credentials on demand.

---

Revision Notes

• Never store secrets in source code.
• HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault are standard tools.
• Dynamic secrets (generated on the fly, auto-expiring) are much safer than static secrets.

---

Further Reading

• HashiCorp Vault Documentation

---

Related Chapters

• Chapter 33 — Managing certificates

---

Last Updated: July 2026