Kubernetes Failure Case Studies

Chapter 54: Kubernetes Failure Case Studies

Learning Objectives

• Debug common Kubernetes failure modes systematically
• Understand why pods get evicted, OOMKilled, or crash-loop
• Diagnose and resolve network, scheduling, and storage failures
• Build resilient Kubernetes deployments that fail gracefully

---

What is this? (Beginner On-Ramp)

Kubernetes failure case studies highlight the immense complexity of modern container orchestration. They detail what happens when the orchestrator itself breaks down—such as nodes dying, networking collapsing, or auto-scaling going rogue.

54.1 Case Study: CrashLoopBackOff

    Symptom: Pod in CrashLoopBackOff, restarts every 30 seconds
    ──────────────────────────────────────────────────────────────

    kubectl get pods -n production
    NAME                        READY   STATUS             RESTARTS
    checkout-api-7df9f-x2kp9   0/1     CrashLoopBackOff   47

    Investigation Workflow:
    ────────────────────────

    Step 1: Get pod events (often the answer)
    kubectl describe pod checkout-api-7df9f-x2kp9 -n production
    Events:
      Warning  BackOff   1m  kubelet  Back-off restarting failed container

    Step 2: Get current container logs
    kubectl logs checkout-api-7df9f-x2kp9 -n production

    Step 3: Get previous container logs (before crash)
    kubectl logs checkout-api-7df9f-x2kp9 -n production --previous
    → Error: FATAL: unable to connect to database: connection refused
    → postgresql://db.internal:5432/appdb

    Step 4: Verify dependency is accessible
    kubectl exec -it checkout-api-7df9f-x2kp9 -n production -- \
        nc -zv db.internal 5432
    → db.internal: Name or service not known

    Root Cause:
    Service was deployed to a new namespace where db.internal
    DNS does not resolve. The correct address is:
    postgres-svc.database.svc.cluster.local

    Fix:
    Update environment variable:
    DB_HOST: postgres-svc.database.svc.cluster.local

    Common CrashLoopBackOff causes:
    ─────────────────────────────────
    ✗ Application can't connect to database/dependencies
    ✗ Missing required environment variables or secrets
    ✗ Crash on startup (bug in initialization code)
    ✗ Wrong container command/entrypoint
    ✗ Permission denied (file system, port binding < 1024 as non-root)
    ✗ OOM on startup (resource limits too low)

---

54.2 Case Study: OOMKilled Pod

    Symptom: Pod keeps restarting with OOMKilled status
    ──────────────────────────────────────────────────────

    kubectl describe pod api-6d89f -n production
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 15 Jan 2024 14:00:12
      Finished:     Mon, 15 Jan 2024 14:03:45

    Memory limit was exceeded → kernel OOM killer terminated process

    Investigation:
    ──────────────
    # Check current resource limits
    kubectl get pod api-6d89f -o jsonpath='{.spec.containers[0].resources}'
    {"limits":{"memory":"512Mi"},"requests":{"memory":"256Mi"}}

    # Check actual memory usage before it dies
    kubectl top pods -n production
    NAME       CPU(cores)   MEMORY(BYTES)
    api-6d89f  120m         498Mi  ← approaching limit!

    # Historical: check Grafana for memory trend
    # container_memory_working_set_bytes{namespace="production"}

    Diagnosis:
    The application needs more than 512Mi during load peaks

    Triage Options:
    ────────────────
    Option A: Increase memory limit (quick fix)
    resources:
      limits:
        memory: "1Gi"    # Increased from 512Mi
      requests:
        memory: "512Mi"  # Also adjust request

    Option B: Find and fix memory leak (correct fix)
    # Profile the application (Chapter 53 GC case study)
    # Heap dump analysis to find unbounded caches

    Option C: Horizontal scaling (distribute load)
    # Add more replicas (share load, each pod uses less memory)
    kubectl scale deployment api --replicas=4

    Alerting Rule to catch before OOMKilled:
    ─────────────────────────────────────────
    # Prometheus rule: Alert when pod is using > 80% of memory limit
    - alert: PodHighMemoryUsage
      expr: |
        container_memory_working_set_bytes / on(pod, namespace)
        (kube_pod_container_resource_limits{resource="memory"}) > 0.8
      for: 10m
      labels: { severity: warning }
      annotations:
        summary: "Pod {{ $labels.pod }} memory > 80% of limit"

---

54.3 Case Study: Pod Pending (Scheduling Failure)

    Symptom: New pods stuck in Pending state
    ──────────────────────────────────────────

    kubectl get pods
    NAME                  READY   STATUS    RESTARTS
    worker-7d9f-xxxx      0/1     Pending   0

    kubectl describe pod worker-7d9f-xxxx
    Events:
      Warning  FailedScheduling  1m  default-scheduler
        0/5 nodes are available:
        2 Insufficient cpu,
        3 node(s) had untolerated taint {node.kubernetes.io/not-ready: NoSchedule}

    Common Pending Causes:
    ────────────────────────

    1. Insufficient Resources (most common)
       kubectl describe nodes | grep -A 5 "Allocated resources"
       → All nodes at 95% CPU requested
       Fix: Add nodes, reduce resource requests, or check for idle pods

    2. Node Affinity / Taint mismatch
       Pod requires: nodeSelector: gpu=true
       No nodes have label gpu=true
       Fix: Add label to node or fix pod spec

    3. PVC not bound (Pod waits for storage)
       kubectl get pvc
       NAME        STATUS    VOLUME
       data-pvc    Pending   (no volume)
       → StorageClass not available or volume quota exceeded

    4. Image pull failure
       Status: ImagePullBackOff (not Pending but looks similar)
       kubectl describe pod → "Failed to pull image: unauthorized"
       Fix: Add imagePullSecrets to pod spec

    Debugging Scheduling:
    ──────────────────────
    # Check node capacity vs requests
    kubectl describe nodes | grep -E "Capacity:|Allocatable:|Requests:"

    # Check node taints
    kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

    # Simulate scheduling
    kubectl get pod <pending-pod> -o yaml | kubectl run --dry-run=server -f -

    # Event-based investigation
    kubectl get events --sort-by='.lastTimestamp' -n production | tail -20

---

54.4 Case Study: Service Not Reachable

    Symptom: Service endpoint returns connection refused / timeout
    ─────────────────────────────────────────────────────────────

    curl http://payment-service.production.svc.cluster.local:8080/health
    curl: (7) Failed to connect: Connection refused

    Step-by-Step Debugging:
    ─────────────────────────

    Step 1: Does the Service exist?
    kubectl get svc -n production payment-service
    Error from server (NotFound): services "payment-service" not found
    → Service doesn't exist! Wrong name or wrong namespace.

    OR if service exists:

    Step 2: Does the Service have endpoints?
    kubectl get endpoints -n production payment-service
    NAME              ENDPOINTS   AGE
    payment-service   <none>      5m
    → No endpoints! Pods not matching selector.

    Step 3: Check selector mismatch
    kubectl describe svc payment-service -n production
    Selector: app=payment-service

    kubectl get pods -n production --show-labels | grep payment
    payment-7d9f-xxx   app=payment,version=v2  ← label is "payment" not "payment-service"!

    Fix: Update selector or pod label to match.

    Step 4: If endpoints exist but still not working — check pod health
    ENDPOINT=$(kubectl get endpoints payment-service -o jsonpath='{.subsets[0].addresses[0].ip}')
    kubectl run debug --image=busybox --rm -it -- wget -O- http://$ENDPOINT:8080/health

    Step 5: NetworkPolicy blocking traffic?
    kubectl get networkpolicies -n production
    # If there's a "deny-all" policy, need explicit allow rule for this traffic

    Service Debugging Flowchart:
    ──────────────────────────────
    Service not working?
    ├── Service exists? (kubectl get svc)
    │   └── No → Create service
    ├── Endpoints populated? (kubectl get ep)
    │   └── No → Check pod labels match service selector
    ├── Pod health ok? (kubectl get pods)
    │   └── No → Fix pod (CrashLoopBackOff or Pending debugging)
    ├── Can reach pod directly?
    │   └── No → Container not listening on expected port
    └── NetworkPolicy blocking?
        └── Yes → Add allow rule

---

54.5 Case Study: etcd Performance Degradation

    Symptom: Kubernetes API server extremely slow, all kubectl commands timeout
    ──────────────────────────────────────────────────────────────────────────

    kubectl get pods  # Takes 30+ seconds

    Investigation:
    ──────────────
    # Check API server metrics
    kubectl get --raw /metrics | grep etcd_request_duration

    # etcd health
    etcdctl endpoint health --endpoints=https://etcd-01:2379
    etcdctl endpoint status --endpoints=https://etcd-01:2379 --write-out=table

    # etcd key count and database size
    etcdctl get "" --prefix --keys-only | wc -l  # Too many keys?
    etcdctl endpoint status -w table              # DB size

    Common Causes:
    ──────────────
    1. etcd disk latency (fsync must be fast — use NVMe, not spinning disk)
       etcdctl check perf              # Benchmarks etcd disk
       # Threshold: disk write latency < 10ms

    2. etcd DB too large (accumulated objects not garbage collected)
       # Large DB = slow etcd
       # Compact and defragment:
       REV=$(etcdctl endpoint status --write-out="json" | jq '.[0].Status.header.revision')
       etcdctl compact $REV
       etcdctl defrag

    3. Too many Kubernetes objects (Events are common offender)
       # Events accumulate fast, default TTL: 1 hour
       # API server flag: --event-ttl=30m  (reduce TTL)

    4. Leader election disk issue
       # etcd writes to disk before committing — disk must be fast
       # Use dedicated NVMe for etcd, separate from other workloads
       # iostat -x 1 → check disk latency on etcd nodes

---

54.6 Kubernetes Debugging Cheat Sheet

# ── Pod Issues ────────────────────────────────────────────
kubectl describe pod <pod>            # Events, resource limits, conditions
kubectl logs <pod> --previous         # Logs before last crash
kubectl exec -it <pod> -- sh          # Shell into container

# ── Service/Network ───────────────────────────────────────
kubectl get endpoints <svc>           # Check if pods are selected
kubectl get networkpolicies           # Check for blocking policies
kubectl run debug --image=nicolaka/netshoot --rm -it -- bash  # Debug pod

# ── Resource Issues ───────────────────────────────────────
kubectl top pods --sort-by=memory     # Pods by memory
kubectl top nodes                     # Node utilization
kubectl describe node <node> | grep -A 10 "Allocated"

# ── Events (the most underused debugging tool) ─────────────
kubectl get events --sort-by='.lastTimestamp' -n <namespace>
kubectl get events --field-selector reason=OOMKilling

# ── Cluster Health ────────────────────────────────────────
kubectl get componentstatuses        # etcd, controller-manager, scheduler
kubectl get nodes -o wide            # Node status and versions

# ── Resource Quota / Limits ───────────────────────────────
kubectl describe resourcequota -n <namespace>
kubectl describe limitrange -n <namespace>

---

54.7 Summary

Failure Mode	Command to Check	Common Fix
CrashLoopBackOff	kubectl logs --previous	Fix app config/dependencies
OOMKilled	kubectl describe pod	Increase memory limit
Pending	kubectl describe pod + events	Add resources, fix affinity
Service unreachable	kubectl get endpoints	Fix label selector
etcd slow	etcdctl check perf	NVMe disk, compact/defrag
Image pull error	kubectl describe pod	Fix imagePullSecrets

---

Next Chapter: Chapter 55: Platform Engineering: IDPs, Golden Paths & Developer Experience

Prerequisites

Chapter 36 (Kubernetes Security).

---

Deep Dive Case Study: The CoreDNS Cascading Failure

In Kubernetes, DNS is the Achilles’ heel. Almost all inter-service communication relies on CoreDNS.

The Incident

A massive spike in traffic caused a backend service to slow down. The frontend service, written in Node.js, had no caching for DNS lookups. Because the backend was slow, the frontend spawned hundreds of additional worker threads, each performing a DNS lookup for the backend service name.

This effectively launched a self-inflicted DDoS attack against the cluster’s CoreDNS pods. CoreDNS CPU maxed out, causing DNS resolution to fail for the entire cluster.

The Impact

Every other service in the cluster lost the ability to resolve names, causing a complete cascading failure of the entire platform, even for services unrelated to the initial traffic spike.

The Fix

1. NodeLocal DNSCache: Deployed a DaemonSet running a local DNS cache on every worker node. Pods query the local cache (via a local IP) instead of crossing the network to hit CoreDNS.
2. Application Level Caching: Fixed the Node.js application to respect DNS TTLs and maintain a connection pool instead of opening a new connection (and doing a new lookup) for every request.
3. CoreDNS Autoscaling: Configured the cluster-proportional-autoscaler to scale CoreDNS replicas dynamically based on the number of nodes/cores in the cluster.

---

Advantages & Disadvantages

Advantages: Exposes the fragility of complex orchestrators, teaches defensive pod design. Disadvantages: Kubernetes evolves so fast that some failure modes get patched out of existence.

---

Common Mistakes

• Not setting CPU/Memory requests and limits, causing node OOM kills.
• Using latest image tags, leading to inconsistent deployments across nodes.
• Failing to configure Readiness and Liveness probes correctly (e.g., Liveness probe that fails when the DB is slow, causing restart loops).

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Pod in CrashLoopBackOff	Application crashing on startup or Liveness probe failing	kubectl logs <pod> --previous; kubectl describe pod <pod>	Fix application error; increase Liveness probe initialDelaySeconds
Nodes going NotReady	Kubelet unresponsive due to CPU/Memory starvation	kubectl describe node <node>; check cloud provider metrics	Enforce resource limits on all pods; ensure kubelet has reserved resources

---

Hands-on Lab

Objective: Debug a failing pod.

# 1. Check pod status and events
# kubectl describe pod <failing-pod>

# 2. Check logs (including previous crashed instance)
# kubectl logs <failing-pod> --previous

# 3. Check node resource usage
# kubectl top nodes

---

Exercises

1. Diagnose a scenario where a deployment has 3 replicas, but 2 are stuck in Pending state. What are the likely causes?
2. Explain the difference between a Readiness probe and a Liveness probe. What happens if a Readiness probe fails? What happens if a Liveness probe fails?

---

Revision Notes

• Resource Limits (limits.memory) enforce a hard cap (OOMKill). Resource Requests (requests.memory) guarantee resources and are used for scheduling.
• CrashLoopBackOff means the container starts and immediately exits.
• Network partition between master and worker nodes causes nodes to become NotReady, and pods may be evicted.
• DNS resolution failures (CoreDNS) are a common cause of internal communication breakdowns in K8s.

---

Further Reading

• Kubernetes Failure Stories (k8s.af)

---

Related Chapters

• Chapter 35 — Container Security

---

Last Updated: July 2026