Chaos Engineering & Reliability Testing

Chapter 43: Chaos Engineering & Reliability Testing

Learning Objectives

Understand the principles and purpose of chaos engineering
Design safe chaos experiments for production
Use Chaos Monkey, Chaos Mesh, and Litmus Chaos
Test failure scenarios: network, disk, CPU, pod, node

What is this? (Beginner On-Ramp)

Chaos Engineering is the practice of intentionally breaking things in production to prove they can survive it. By safely simulating failures (like pulling the plug on a database), engineers find hidden weaknesses before they turn into real, uncontrolled outages.

30.1 What is Chaos Engineering?

    Chaos Engineering Philosophy
    ─────────────────────────────

    Traditional Testing:
    "Does our system work correctly under normal conditions?"

    Chaos Engineering:
    "Does our system stay resilient when things go wrong?"

    The Core Insight:
    ──────────────────
    Distributed systems have failures constantly in production.
    If you don't test for failures, your first failure test
    happens during an incident, in front of customers.

    Better to:
    1. Hypothesize how the system should behave under failure
    2. Inject controlled failures in staging/production
    3. Observe actual behavior
    4. Fix gaps between hypothesis and reality

    Famous origin: Netflix Chaos Monkey (2011)
    "We were moving to AWS. We needed to know if our services
     could survive instance failures. So we started randomly
     killing instances."

30.2 The Scientific Method of Chaos

    Chaos Experiment Design
    ────────────────────────

    Step 1: Define the steady state
    "Under normal conditions, checkout success rate > 99.9%,
     P99 latency < 200ms, and error rate < 0.1%"

    Step 2: Form a hypothesis
    "If one of three checkout service replicas dies,
     the system will recover within 30 seconds and
     maintain the steady state"

    Step 3: Plan the experiment
    "Kill one replica of checkout-service during off-peak hours
     on a Tuesday morning"

    Step 4: Run with minimal blast radius
    "Start with staging. Then 5% of prod traffic. Then wider."

    Step 5: Observe
    "Monitor: pod count, error rate, latency, client-side metrics"

    Step 6: Analyze
    "Recovery took 45 seconds (too slow). Latency spiked to 400ms.
     Hypothesis was wrong — pod scheduling is too slow."

    Step 7: Fix and repeat
    "Add Pod Disruption Budgets, pre-pull images, add liveness probes"

30.3 Common Chaos Experiments

    Chaos Experiment Catalog
    ─────────────────────────

    Infrastructure Failures:
    ✦ Pod kill (Kubernetes) — Does auto-recovery work?
    ✦ Node kill — Does workload reschedule within SLO?
    ✦ Zone failure — Multi-AZ failover working?
    ✦ Region failure — DR tested?

    Network Failures:
    ✦ Network latency injection — Can services handle slow dependencies?
    ✦ Packet loss — Does retry logic work correctly?
    ✦ DNS failure — Caching working? Fallbacks?
    ✦ Connection blackhole — Do timeouts trigger?

    Resource Pressure:
    ✦ CPU stress — Does autoscaling trigger in time?
    ✦ Memory pressure — Does OOM killer behave correctly?
    ✦ Disk filling — Does the system degrade gracefully?

    Dependency Failures:
    ✦ Database down — Do circuit breakers open? Graceful degradation?
    ✦ Cache miss — Can the system handle 100% cache miss rate?
    ✦ Third-party API timeout — Do timeouts and fallbacks work?

    Application Failures:
    ✦ Slow consumers — Does backpressure work correctly?
    ✦ Memory leak — Does the system detect and restart?

30.4 Chaos Mesh (Kubernetes)

# Chaos Mesh — CNCF sandbox project for Kubernetes chaos

# Install
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh --namespace=chaos-testing --create-namespace

# ── Pod Kill Experiment ────────────────────────────────────
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: kill-checkout-pod
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one                    # Kill one pod at a time
  selector:
    namespaces: [production]
    labelSelectors:
      app: checkout-service
  scheduler:
    cron: "@every 5m"           # Kill one pod every 5 minutes

---
# ── Network Latency Injection ──────────────────────────────
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: checkout-to-payment-latency
spec:
  action: delay
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: checkout-service
  delay:
    latency: "500ms"            # Add 500ms to all egress traffic
    correlation: "25"           # 25% packet correlation
    jitter: "100ms"             # ±100ms jitter
  direction: to
  target:                       # Only affect traffic to payment service
    selector:
      namespaces: [production]
      labelSelectors:
        app: payment-service
  duration: "5m"

---
# ── CPU Stress ─────────────────────────────────────────────
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: cpu-stress-checkout
spec:
  mode: one
  selector:
    namespaces: [production]
    labelSelectors:
      app: checkout-service
  stressors:
    cpu:
      workers: 2               # 2 CPU stress workers
      load: 80                 # 80% CPU load
  duration: "10m"

---
# ── Disk IO Latency ────────────────────────────────────────
apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: disk-latency-postgres
spec:
  action: latency
  mode: one
  selector:
    namespaces: [production]
    labelSelectors:
      app: postgres
  volumePath: /var/lib/postgresql
  delay: "100ms"               # Add 100ms to all disk I/O
  percent: 50                  # Affect 50% of I/O operations
  duration: "5m"

30.5 Linux-Level Chaos Tools

# ── tc (traffic control) — Network Chaos ─────────────────
# Add 100ms latency to eth0
tc qdisc add dev eth0 root netem delay 100ms

# Add 100ms ± 20ms jitter
tc qdisc add dev eth0 root netem delay 100ms 20ms

# Add 10% packet loss
tc qdisc add dev eth0 root netem loss 10%

# Add packet corruption
tc qdisc add dev eth0 root netem corrupt 5%

# Add bandwidth limit (1Mbps)
tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms

# Remove all tc rules
tc qdisc del dev eth0 root

# ── CPU Stress ────────────────────────────────────────────
# stress-ng: advanced stress testing
stress-ng --cpu 4 --cpu-load 80 --timeout 60s
stress-ng --vm 2 --vm-bytes 1G --timeout 60s      # Memory pressure
stress-ng --io 4 --timeout 60s                    # I/O stress
stress-ng --all 0 --timeout 60s                   # Everything

# ── Disk Fill ─────────────────────────────────────────────
# Fill disk with junk (CAREFUL! can crash the system)
dd if=/dev/zero of=/tmp/diskfill bs=1M count=1000  # Fill 1GB
# Clean up:
rm /tmp/diskfill

# Better: use fallocate (instant, no I/O)
fallocate -l 5G /tmp/5gb-test
rm /tmp/5gb-test

# ── Kill a Process Unexpectedly ───────────────────────────
# SIGKILL (instant, no cleanup)
kill -9 $(pidof myapp)

# SIGSEGV (simulate crash)
kill -11 $(pidof myapp)

# ── Memory Pressure ───────────────────────────────────────
# Consume memory until OOM
python3 -c "l=[]; [l.append('x'*1024*1024) for _ in range(10000)]"

# ── Network Partition (block specific host) ───────────────
# Block traffic to/from the database
iptables -A INPUT -s 10.0.1.5 -j DROP    # Block incoming from DB
iptables -A OUTPUT -d 10.0.1.5 -j DROP  # Block outgoing to DB
# Restore:
iptables -D INPUT -s 10.0.1.5 -j DROP
iptables -D OUTPUT -d 10.0.1.5 -j DROP

30.6 Chaos Experiment Safety Guidelines

    Safety Rules for Chaos Engineering
    ────────────────────────────────────

    1. Start in staging, never production first
       Prove the experiment is safe and instrumented

    2. Define rollback plan BEFORE running
       Know exactly how to stop the experiment

    3. Use circuit breakers (stop conditions)
       "If error rate > 5%, abort the experiment"

    4. Run during low-traffic periods
       Reduces customer impact if hypothesis is wrong

    5. Never run two chaos experiments simultaneously
       Cannot correlate cause and effect

    6. Observe actively during the experiment
       Have engineers watching dashboards in real-time

    7. Document results regardless of outcome
       Failed hypotheses are valuable data

    8. Get stakeholder buy-in
       SRE, product, and leadership should know when running in prod

    9. Start with blast radius = 1 pod, not all pods
       Mode: one (not all) in Chaos Mesh

    10. Have a "break glass" stop mechanism
        One command to restore normal state immediately

30.7 Interview Questions

Q1: What is chaos engineering and what problem does it solve?

Answer: Chaos engineering is the practice of intentionally introducing controlled failures into a system to verify its resilience. The problem it solves: distributed systems are complex and have many potential failure modes (node crashes, network partitions, disk failures, dependency timeouts). If these failure modes are never tested, the first time they’re encountered is during a real incident — at maximum customer impact. Chaos engineering shifts failure testing to controlled experiments during low-traffic periods, allowing teams to discover and fix gaps between how they think the system behaves under failure and how it actually behaves. Famous example: Netflix’s Chaos Monkey kills random EC2 instances in production — this forced their engineering to build services that are resilient to instance failure.

Q2: How do you design a safe chaos experiment?

Answer: (1) Define steady state: Establish the metrics that should remain stable (error rate < 0.1%, P99 < 200ms). (2) Form a hypothesis: “If X fails, Y should happen within Z seconds.” (3) Start minimal: Begin with mode: one (one pod), not mode: all. Start in staging. (4) Define abort conditions: “If error rate exceeds 5%, stop immediately.” (5) Plan rollback: Know exactly how to restore normal state before starting. (6) Run during low traffic: Tuesday 10am, not Friday 5pm. (7) Actively monitor: Engineers watching dashboards during the experiment. (8) Run one experiment at a time: Can’t diagnose if two failures are happening simultaneously.

30.8 Summary

Tool	Platform	Use Case
Chaos Monkey	Netflix OSS	Random instance termination
Chaos Mesh	Kubernetes	Pod, network, IO, stress chaos
Litmus Chaos	Kubernetes	Workflow-based chaos experiments
tc/netem	Linux	Network latency/loss/corruption
stress-ng	Linux	CPU/memory/IO stress

Next Chapter: Chapter 31: Capacity Planning & Performance Tuning

Prerequisites

Chapter 38 (High Availability), Chapter 41 (SRE).

Advantages & Disadvantages

Advantages: Proves resilience empirically, trains the team for real incidents (Game Days). Disadvantages: Can cause actual production outages if blast radius isn’t contained.

Common Mistakes

Running chaos experiments in production without testing in staging first.
Running experiments without a clear hypothesis or baseline metrics.
Causing a major outage because the ‘blast radius’ wasn’t contained.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Chaos experiment causes unrecoverable failure	Blast radius too large or missing abort condition	`Review experiment design`	Implement automated stop conditions (e.g., if error rate > 5%, stop experiment)
Experiment yields no insights	System wasn’t under realistic load	`Review traffic metrics during experiment`	Ensure simulated or real traffic is flowing during the test

Hands-on Lab

Objective: Introduce controlled chaos.

# 1. Use stress-ng to simulate a CPU spike on a test VM
stress-ng --cpu 2 --timeout 60s

# 2. Simulate network latency using tc (Traffic Control)
# sudo tc qdisc add dev eth0 root netem delay 100ms
# To remove:
# sudo tc qdisc del dev eth0 root

Exercises

Design a chaos experiment to test what happens when your database primary node fails. What is the hypothesis?
Use tc to inject 5% packet loss on a test VM and observe how it affects application latency using ping and curl.

Revision Notes

Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions.
Always define a steady state, form a hypothesis, and minimize the blast radius.
Start in staging; graduate to production only when confident.
Game Days are planned team exercises to practice incident response using chaos injection.