PKI, TLS, mTLS & Certificate Management

Chapter 33: PKI, TLS, mTLS & Certificate Management

Learning Objectives

• Understand PKI hierarchy and certificate chains
• Configure TLS correctly for production
• Implement mTLS for service-to-service authentication
• Automate certificate lifecycle with cert-manager and Vault PKI

---

What is this? (Beginner On-Ramp)

PKI (Public Key Infrastructure) and TLS (Transport Layer Security) are the technologies that encrypt the internet (the ‘s’ in https). Mutual TLS (mTLS) takes it a step further: not only does the client verify the server is legit, the server also verifies the client.

33.1 PKI Architecture

    Public Key Infrastructure (PKI) Hierarchy
    ───────────────────────────────────────────

    Root CA (offline, air-gapped)
        │ Signs
        ▼
    Intermediate CA (online)
        │ Signs
        ▼
    Leaf Certificate (your server/service)

    Why the hierarchy?
    • Root CA is self-signed — if compromised, everything is compromised
    • Keep Root CA offline (HSM, safe)
    • Intermediate CA does day-to-day signing
    • If Intermediate is compromised → revoke it, re-issue from Root
    • Leaf certs are short-lived (90 days) → minimize damage if leaked

    Certificate Contents (X.509):
    ──────────────────────────────
    Subject:       CN=api.example.com, O=ACME Corp
    Issuer:        CN=Intermediate CA
    Valid From:    2024-01-01
    Valid Until:   2024-04-01 (90 days)
    Public Key:    RSA-2048 / ECDSA-P256 / Ed25519
    SANs:          api.example.com, *.api.example.com
    Key Usage:     Digital Signature, Key Encipherment
    Extensions:    CRL Distribution Points, OCSP URL

---

33.2 TLS Handshake (TLS 1.3)

    TLS 1.3 Handshake (simplified)
    ────────────────────────────────

    Client                              Server
    ──────                              ──────
    ClientHello
    (supported ciphers, key_share) ───────────►

                                        ServerHello
                                        (chosen cipher, key_share)
                                        Certificate (server cert)
                          ◄─────────── CertificateVerify, Finished

    Both derive session keys from key_share (ECDH)
    All further traffic is encrypted

    Client verifies:
    1. Certificate chain leads to trusted Root CA
    2. Certificate CN/SAN matches hostname
    3. Certificate not expired
    4. Certificate not revoked (OCSP/CRL)

    Finished ─────────────────────────►

    HTTP/2 request ───────────────────►
                          ◄─────────── HTTP/2 response

    TLS 1.3 improvements over 1.2:
    • 1-RTT handshake (vs 2-RTT)
    • 0-RTT resumption (session resumption)
    • Removed weak cipher suites (RC4, DES, SHA-1)
    • Forward secrecy mandatory (ECDHE only)

---

33.3 Nginx TLS Configuration (Production-Hardened)

/etc/nginx/sites-available/api.example.com

server {
    listen 443 ssl http2;
    server_name api.example.com;

    # ── Certificates ──────────────────────────────────────
    ssl_certificate     /etc/ssl/certs/api.example.com.crt;
    ssl_certificate_key /etc/ssl/private/api.example.com.key;

    # ── Protocols (TLS 1.2 minimum, 1.3 preferred) ────────
    ssl_protocols TLSv1.2 TLSv1.3;

    # ── Cipher Suites ─────────────────────────────────────
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
    ssl_prefer_server_ciphers off;   # Let client choose (TLS 1.3)

    # ── Session Resumption ────────────────────────────────
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;         # Disable tickets (forward secrecy)

    # ── OCSP Stapling ─────────────────────────────────────
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/ssl/certs/ca-chain.crt;
    resolver 8.8.8.8 8.8.4.4 valid=300s;

    # ── HSTS ──────────────────────────────────────────────
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    # ── Other Security Headers ────────────────────────────
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;
    add_header Content-Security-Policy "default-src 'self'" always;
}

# Redirect HTTP → HTTPS
server {
    listen 80;
    server_name api.example.com;
    return 301 https://$host$request_uri;
}

---

33.4 mTLS: Mutual TLS

mTLS (mutual TLS) adds client certificate authentication — both sides verify each other.

    Standard TLS:
    Client verifies Server certificate → "Is this really api.example.com?"

    mTLS:
    Client verifies Server certificate → "Is this really api.example.com?"
    Server verifies Client certificate → "Is this really the checkout service?"

    Use Cases:
    ✓ Service-to-service auth (microservices)
    ✓ API authentication (instead of API keys)
    ✓ Zero Trust network model
    ✓ Machine-to-machine communication

# mTLS Server Config
server {
    listen 443 ssl http2;

    ssl_certificate     /etc/ssl/server.crt;
    ssl_certificate_key /etc/ssl/server.key;

    # Require and verify client certificate
    ssl_client_certificate /etc/ssl/ca/ca-chain.crt;
    ssl_verify_client on;
    ssl_verify_depth 2;

    # Pass client cert info to upstream
    proxy_set_header X-SSL-Client-CN    $ssl_client_s_dn_cn;
    proxy_set_header X-SSL-Client-Cert  $ssl_client_escaped_cert;

    location / {
        proxy_pass http://backend;
    }
}

# Generate client certificate for a service
# CA key (already exists)
openssl genrsa -out checkout-service.key 2048
openssl req -new -key checkout-service.key \
    -subj "/CN=checkout-service/O=production" \
    -out checkout-service.csr

# Sign with CA
openssl x509 -req -in checkout-service.csr \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -out checkout-service.crt -days 90 \
    -extensions v3_req

# Use client cert in curl
curl --cert checkout-service.crt --key checkout-service.key \
     https://payment-service.internal/charge

---

33.5 cert-manager (Kubernetes)

cert-manager automates certificate issuance and renewal in Kubernetes.

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

# ClusterIssuer using Let's Encrypt
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-token
            key: api-token

---
# Request a certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: production
spec:
  secretName: api-tls-secret        # Where cert is stored
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - api.example.com
    - www.example.com
  duration: 2160h                    # 90 days
  renewBefore: 720h                  # Renew 30 days before expiry

---
# Internal CA issuer (for service-to-service mTLS)
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret   # Root CA cert stored as secret

---

33.6 Certificate Lifecycle Management

# ── View Certificate ───────────────────────────────────────
openssl x509 -in cert.crt -text -noout | grep -E "Subject:|Issuer:|Not After:|DNS:"

# Quick expiry check:
openssl x509 -enddate -noout -in cert.crt

# Check remote server certificate:
echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
    | openssl x509 -noout -subject -enddate

# ── Certificate Monitoring ────────────────────────────────
# Prometheus blackbox exporter:
# ssl_cert_not_after metric → alert if expiring within 30 days

# Alert rule:
# - alert: CertificateExpiringSoon
#   expr: (ssl_cert_not_after - time()) / 86400 < 30
#   for: 1h
#   annotations:
#     summary: "Certificate for {{ $labels.target }} expires in {{ $value | humanize }} days"

# ── Revocation: OCSP ──────────────────────────────────────
# Check if a certificate is revoked:
openssl ocsp -issuer issuer.crt -cert server.crt \
    -url http://ocsp.example.com -resp_text

# ── Certificate Pinning (advanced) ────────────────────────
# Pin certificate public key hash (for high-security APIs)
openssl x509 -pubkey -noout < cert.crt | \
    openssl pkey -pubin -outform der | \
    openssl dgst -sha256 -binary | base64
# → Use this hash in HPKP header or in app config

---

33.7 Interview Questions

Q1: What is the difference between TLS and mTLS?

Answer: Standard TLS provides one-way authentication: the client verifies the server’s identity (via certificate signed by a trusted CA) but the server doesn’t verify the client. This is what HTTPS does — your browser verifies that github.com is really GitHub, but GitHub doesn’t verify who you are at the TLS layer (it uses passwords/tokens instead). mTLS (mutual TLS) adds client certificate authentication: both the client and server present certificates, and both verify each other. This provides cryptographic, mutual authentication without passwords. Use cases: service-to-service in microservices (checkout service proves its identity to payment service), zero trust networking, machine-to-machine APIs.

Q2: Why use a certificate hierarchy (Root CA → Intermediate CA → Leaf) instead of one CA?

Answer: The Root CA’s private key is the ultimate trust anchor — if it’s compromised, ALL certificates it ever issued must be considered untrusted. The hierarchy limits blast radius: (1) Root CA is kept offline (air-gapped, in HSM) and is NEVER used for day-to-day signing. (2) Intermediate CA is online and does daily certificate issuance. (3) If the Intermediate CA is compromised, revoke it and issue a new Intermediate from the Root — without touching the Root. Leaf certificates are short-lived (90 days) to minimize damage from key exposure. All major CAs (Let’s Encrypt, DigiCert) use this hierarchy.

---

33.8 Summary

Concept	Key Point
Root CA	Offline, ultimate trust anchor
Intermediate CA	Online, signs leaf certs
Leaf cert	Short-lived (90d), one per service
TLS 1.3	1-RTT, mandatory forward secrecy
mTLS	Both sides present certificates
OCSP stapling	Server proves cert validity inline
cert-manager	Automates k8s cert lifecycle

---

Next Chapter: Chapter 34: Secrets Management: HashiCorp Vault, KMS & HSM

Prerequisites

Basic cryptography concepts.

---

Advantages & Disadvantages

Advantages: Mathematical guarantee of identity and privacy, stops man-in-the-middle attacks. Disadvantages: Certificate expiration outages are common, managing CAs is an operational burden.

---

Common Mistakes

• Using self-signed certificates in production without a proper internal CA.
• Hardcoding certificate expiration dates, leading to outages when they expire.
• Not checking certificate revocation lists (CRLs) or OCSP.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
TLS Handshake failure	Certificate expired, mismatched hostname, or untrusted CA	openssl s_client -connect host:443	Renew cert, fix SANs, or distribute CA root to client
mTLS connection denied	Client certificate missing or not signed by trusted CA	Check server logs for 'bad certificate'	Provide valid client cert in the request

---

Hands-on Lab

Objective: Inspect a TLS certificate.

# 1. Fetch and inspect a remote certificate
echo | openssl s_client -connect google.com:443 2>/dev/null | openssl x509 -noout -dates -subject -issuer

# 2. Generate a local CSR (Certificate Signing Request)
openssl req -new -newkey rsa:2048 -nodes -keyout mydomain.key -out mydomain.csr

---

Exercises

1. Set up a local Certificate Authority (CA) using cfssl or openssl, generate a server certificate, and configure Nginx to use it.
2. Configure Nginx to require mutual TLS (mTLS) by verifying client certificates against your local CA.

---

Revision Notes

• PKI relies on a chain of trust back to a Root CA.
• TLS provides encryption (privacy) and authentication (identity).
• mTLS (Mutual TLS) requires both the client and server to authenticate each other using certificates.

---

Further Reading

• man openssl
• Let’s Encrypt documentation

---

Related Chapters

• Chapter 34 — Managing the private keys

---

Last Updated: July 2026