Container & Kubernetes Security

Chapter 35: Container & Kubernetes Security

Learning Objectives

• Understand container security at every layer (image → runtime → cluster)
• Harden Docker container builds and runtime security
• Apply Kubernetes security controls (PSA, NetworkPolicy, RBAC)
• Scan images for vulnerabilities and implement supply chain security

---

What is this? (Beginner On-Ramp)

Container security is the practice of locking down Docker and Kubernetes environments. Because containers share the same underlying operating system, if one container gets hacked, a poor security configuration could allow the hacker to take over the whole machine.

32.1 Container Threat Model

    Container Security Layers
    ──────────────────────────

    ┌────────────────────────────────────────────────────────────┐
    │  1. Supply Chain                                            │
    │     Base image vulnerabilities, malicious packages,        │
    │     compromised build pipelines                            │
    ├────────────────────────────────────────────────────────────┤
    │  2. Image Build                                             │
    │     Secrets in layers, running as root, large attack       │
    │     surface (unnecessary packages/tools)                   │
    ├────────────────────────────────────────────────────────────┤
    │  3. Runtime                                                 │
    │     Privileged containers, host namespace sharing,         │
    │     dangerous capabilities, excessive permissions          │
    ├────────────────────────────────────────────────────────────┤
    │  4. Kubernetes Cluster                                      │
    │     RBAC misconfiguration, open API server, network        │
    │     policies missing, secrets in plaintext                 │
    ├────────────────────────────────────────────────────────────┤
    │  5. Node                                                    │
    │     Container escape → node compromise → cluster control   │
    └────────────────────────────────────────────────────────────┘

---

32.2 Secure Docker Images

# ── Secure Dockerfile Best Practices ──────────────────────

# 1. Use minimal base images (smaller attack surface)
FROM gcr.io/distroless/nodejs:18          # Google distroless: no shell, no apt
FROM alpine:3.19                           # Alpine: minimal with musl libc
# AVOID: FROM ubuntu:latest, FROM centos:7 (too large, too many packages)

# 2. Multi-stage build (no build tools in production image)
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production              # Only prod dependencies
COPY . .
RUN npm run build

# Production stage
FROM gcr.io/distroless/nodejs:20          # No shell, no package manager
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 8080
USER nonroot                               # Non-root user (uid=65532 in distroless)
CMD ["dist/server.js"]

# 3. Never embed secrets in Dockerfiles or images!
# BAD:
# ENV DB_PASSWORD=secret123

# GOOD: Pass at runtime
# docker run -e DB_PASSWORD=$SECRET myapp
# Or use Docker secrets / Kubernetes secrets

# 4. Pin exact versions (not :latest)
FROM node:20.11.1-alpine3.19             # Pin exact version

# 5. Verify base image integrity (in CI)
# Use digest pinning:
FROM node@sha256:abc123...               # Immutable reference

# 6. Run as non-root
RUN addgroup -g 1001 -S appgroup && \
    adduser -u 1001 -S appuser -G appgroup
USER appuser

# 7. Read-only filesystem
# (Enforced at runtime, but prepare your app for it)
# Ensure writes go to /tmp or dedicated volumes

---

32.3 Image Scanning

# ── Trivy: Container Image Scanning ──────────────────────
# Install
brew install trivy         # macOS
apt-get install trivy      # Linux

# Scan an image
trivy image nginx:latest

# Scan only CRITICAL and HIGH vulnerabilities
trivy image --severity HIGH,CRITICAL nginx:latest

# Output as JSON for CI integration
trivy image --format json -o results.json nginx:latest

# Scan a Dockerfile
trivy config Dockerfile

# Scan Kubernetes manifests for misconfigurations
trivy k8s --report summary cluster

# ── CI Integration (GitHub Actions) ───────────────────────
# .github/workflows/security.yml
name: Security Scan
on: [push, pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .
      - name: Run Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          severity: CRITICAL,HIGH
          exit-code: '1'            # Fail CI on vulnerabilities
      - name: Upload results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: trivy-results.sarif

# ── Docker Content Trust ──────────────────────────────────
# Sign and verify images (Notary)
export DOCKER_CONTENT_TRUST=1         # Enable DCT
docker push myregistry.io/myapp:v1.0  # Signs the image

# ── SBOM (Software Bill of Materials) ────────────────────
# Generate SBOM for compliance
trivy image --format cyclonedx --output sbom.json nginx:latest
syft nginx:latest -o cyclonedx-json > sbom.json

---

32.4 Kubernetes Pod Security

# Kubernetes Pod Security Standards (PSA)
# Replaces deprecated PodSecurityPolicies

# 3 levels:
# privileged: Unrestricted (only for infra workloads)
# baseline:   Minimum restrictions (prevents worst misconfigs)
# restricted: Hardened (requires non-root, no host access)

# Apply to namespace via labels:
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted      # Block non-compliant pods
    pod-security.kubernetes.io/audit: restricted        # Audit non-compliant
    pod-security.kubernetes.io/warn: restricted         # Warn on non-compliant

---
# Fully hardened pod specification
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  # Pod-level security context
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault           # Enable seccomp
    supplementalGroups: []

  # Don't auto-mount service account token if not needed
  automountServiceAccountToken: false

  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      allowPrivilegeEscalation: false  # No setuid/setgid binaries
      readOnlyRootFilesystem: true     # Immutable container FS
      capabilities:
        drop: ["ALL"]                  # Drop ALL capabilities
        add: []                        # Add back only if truly needed
      seccompProfile:
        type: RuntimeDefault

    # Resource limits (also a security control)
    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
      requests:
        cpu: "100m"
        memory: "128Mi"

    # Write-allowed volumes when readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: app-data
      mountPath: /app/data

  volumes:
  - name: tmp
    emptyDir: {}
  - name: app-data
    emptyDir: {}

---

32.5 Kubernetes Network Policies

# Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}          # Applies to all pods
  policyTypes:
  - Ingress
  - Egress

---
# Allow only specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-checkout-traffic
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: checkout-service

  policyTypes:
  - Ingress
  - Egress

  ingress:
  # Only allow ingress from API gateway
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway
    ports:
    - protocol: TCP
      port: 8080

  egress:
  # Allow to payment service
  - to:
    - podSelector:
        matchLabels:
          app: payment-service
    ports:
    - protocol: TCP
      port: 8081
  # Allow to PostgreSQL
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53

---

32.6 Kubernetes RBAC

# Principle of Least Privilege for Service Accounts

# 1. Create dedicated service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/checkout-service-role

---
# 2. Create minimal role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: checkout-service-role
  namespace: production
rules:
# Only what checkout needs
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
  resourceNames: ["checkout-config"]        # Specific resource only!

- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["checkout-db-secret"]     # Specific secret only!

---
# 3. Bind role to service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: checkout-service-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: checkout-service
  namespace: production
roleRef:
  kind: Role
  name: checkout-service-role
  apiGroup: rbac.authorization.k8s.io

---
# 4. Audit RBAC (who can do what)
# Check what a service account can do:
kubectl auth can-i --list --as=system:serviceaccount:production:checkout-service

# Check if specific action is allowed:
kubectl auth can-i get pods --as=system:serviceaccount:production:checkout-service

---

32.7 Runtime Security with Falco

# Falco: Runtime security monitoring for containers
# Detects anomalous behavior using kernel events

# Install
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco --namespace falco --create-namespace

# Custom rules: /etc/falco/rules.d/custom_rules.yaml
- rule: Terminal Shell in Container
  desc: A shell was used as the entrypoint/exec point of a container
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
  output: >
    Container started with a shell (evt_type=%evt.type
    container=%container.id image=%container.image.repository
    user=%user.name command=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, T1059.004]

- rule: Write to sensitive directory in container
  desc: Detect writes to sensitive directories in a container
  condition: >
    open_write and container and not root_dir
    and (fd.name startswith /etc or fd.name startswith /root)
  output: >
    Write to sensitive dir (user=%user.name
    command=%proc.cmdline file=%fd.name container=%container.id)
  priority: ERROR

- rule: Outbound Connection to Unexpected Port
  desc: Container making outbound connection to non-standard port
  condition: >
    outbound and container
    and not (fd.sport in (80, 443, 5432, 6379, 9092))
  output: >
    Unexpected outbound connection (command=%proc.cmdline
    connection=%fd.name container=%container.id)
  priority: WARNING

---

32.8 Interview Questions

Q1: What does “distroless” mean and why use distroless base images?

Answer: Distroless images (from Google) contain only the application and its runtime dependencies — no shell, no package manager (apt/yum), no debugging tools. This dramatically reduces the attack surface: if an attacker achieves code execution in a container, they cannot run bash, install tools, or use the package manager to escalate. Container escape exploits often rely on having shell access to execute subsequent commands. Additionally, smaller images mean fewer packages, fewer CVEs, faster pull times, and smaller storage. The trade-off: debugging is harder (no exec into the container). For debugging, use ephemeral debug containers: kubectl debug -it pod-name --image=busybox.

Q2: What is a Kubernetes NetworkPolicy and what happens if you don’t configure one?

Answer: A NetworkPolicy controls which pods can communicate with which other pods and external endpoints. Without NetworkPolicies, Kubernetes has a default-allow model — any pod can talk to any other pod in the cluster on any port. This means: if an attacker compromises your frontend pod, they can directly reach your database pod. NetworkPolicies implement microsegmentation: checkout service can only talk to payment service and PostgreSQL, not to any other service. Important: NetworkPolicies require a CNI plugin that supports them (Calico, Cilium, Weave) — vanilla flannel does NOT enforce NetworkPolicies. Always apply a default-deny-all policy first, then explicitly allow required traffic.

---

32.9 Summary

Control	Tool/Method
Image scanning	Trivy, Snyk
Image signing	Sigstore, Cosign
Runtime security	Seccomp, AppArmor, Falco
Pod security	PodSecurityContext, PSA
Network isolation	NetworkPolicy (Calico/Cilium)
Access control	RBAC + dedicated ServiceAccounts
Secrets	External Secrets Operator, Vault

---

Next Chapter: Chapter 33: Secrets Management & PKI

Prerequisites

Chapter 5 (Cgroups/Namespaces), Docker basics.

---

Deep Dive: Distroless Images

When building containers, the base image you choose heavily dictates your attack surface.

The Problem with Alpine and Ubuntu

Even minimal distributions like Alpine Linux contain package managers (apk), shells (/bin/sh), and utilities (wget, curl). If an attacker finds a Remote Code Execution (RCE) vulnerability in your Node.js app, they can drop into a shell and use curl to download a cryptominer.

The Distroless Solution

“Distroless” images (popularized by Google) contain only your application and its runtime dependencies. They do not contain package managers, shells, or standard Unix utilities.

# Multi-stage build using Distroless
FROM golang:1.20 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp main.go

# The final image uses a distroless base
FROM gcr.io/distroless/base-debian11
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

If an attacker achieves RCE in a distroless container, they cannot execute sh or curl because those binaries simply do not exist in the filesystem. This effectively stops most automated exploit scripts dead in their tracks.

---

Advantages & Disadvantages

Advantages: Immutability makes patching easier, tools like Trivy catch CVEs early. Disadvantages: Shared kernel means a single breakout compromises the whole host.

---

Common Mistakes

• Running containers as root (USER root in Dockerfile).
• Mounting the docker socket (/var/run/docker.sock) into a container (allows container escape).
• Using latest image tags, making vulnerability scanning and reproducibility impossible.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Container killed unexpectedly	OOM killed or security profile violation (AppArmor/Seccomp)	`docker inspect <container_id>	grep OOMKilled`
Vulnerability scanner reports critical CVEs	Outdated base image or vulnerable dependencies	trivy image myapp:latest	Update base image (e.g., switch to alpine or distroless) and rebuild

---

Hands-on Lab

Objective: Scan a container image.

# 1. Install Trivy (if not present)
# curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh

# 2. Scan a public image
# trivy image nginx:latest

# 3. Check container privileges
docker run --rm -it alpine sh -c 'id; capsh --print'

---

Exercises

1. Write a secure Dockerfile for a Node.js app that uses a non-root user, copies only necessary files, and uses a minimal base image.
2. Run a container with --privileged. Demonstrate how you can view the host’s devices (ls /dev) from inside the container, proving it’s insecure.

---

Revision Notes

• Containers are not VMs; they share the host kernel.
• Security relies on Namespaces, Cgroups, Seccomp, Capabilities, and MAC.
• Always scan images for CVEs in the CI/CD pipeline.
• Use minimal base images (Distroless, Alpine) to reduce the attack surface.

---

Further Reading

• CIS Docker Benchmark
• Container Security by Liz Rice

---

Related Chapters

• Chapter 36 — Kubernetes Security

---

Last Updated: July 2026