Benchmarking, Capacity Planning & Bottleneck Analysis

Chapter 26: Benchmarking, Capacity Planning & Bottleneck Analysis

Learning Objectives

Design and run proper benchmarks without common pitfalls
Apply Amdahl’s Law and the Universal Scalability Law
Build a capacity planning model from real data
Identify bottlenecks using queueing theory intuition

What is this? (Beginner On-Ramp)

Benchmarking is running controlled stress tests on a system to see how fast it really is. By simulating heavy traffic, you can find out the maximum capacity of your server and discover what breaks first when the system is pushed to its limits.

26.1 Benchmarking Principles

    Common Benchmarking Mistakes
    ─────────────────────────────

    1. Measuring cold start only
       → System hasn't warmed up caches
       → Run benchmark twice: discard first run

    2. Not controlling for variance
       → Run at least 3-5 times, report mean + stddev
       → Single run results are meaningless

    3. Benchmarking the wrong thing
       → Microbenchmark shows function is 2x faster
       → End-to-end performance unchanged (it's not the bottleneck)

    4. Not matching production conditions
       → Benchmarking with 1 thread in dev, production has 100
       → Benchmarking with empty database, production has 50M rows

    5. Ignoring coordinated omission
       → Load generator pauses when server is slow
       → Doesn't simulate real traffic (which doesn't pause)
       → Use: wrk2, HDR histogram for correct latency measurement

    6. Tuning for the benchmark, not production
       → Parameters that make benchmark look good hurt real users

26.2 Load Testing Tools

# ── wrk2: Correct Latency Measurement ────────────────────
# wrk2 avoids coordinated omission (unlike wrk)
wrk2 -t8 -c400 -d30s -R 10000 --latency http://api.example.com/
# -R: constant request rate (10000 RPS)
# Output: correct percentile latency histogram

# ── k6: Full Scenario Testing ─────────────────────────────
# See Chapter 31 for full k6 coverage

# ── Gatling: JVM-based, good for complex scenarios ────────
# Define simulations in Scala DSL
# Built-in HTML reports with percentile charts

# ── Apache Benchmark (quick sanity check) ─────────────────
ab -n 100000 -c 200 http://localhost:8080/api/health
# Good for: quick test of single endpoint
# Bad for: realistic load, latency accuracy

# ── Database-specific ─────────────────────────────────────
# pgbench (PostgreSQL):
pgbench -i -s 100 bench_db         # Initialize 100-scale database (~1.5GB)
pgbench -c 50 -j 10 -T 300 bench_db  # 50 clients, 10 threads, 5 min
# Output: TPS, latency average and stddev

# sysbench (MySQL/MariaDB):
sysbench --db-driver=mysql oltp_read_write \
    --threads=16 --time=60 run

26.3 Amdahl’s Law & Scalability Limits

    Amdahl's Law
    ─────────────
    If p fraction of a program can be parallelized:

    Speedup = 1 / ((1 - p) + p/N)

    Where N = number of processors/threads

    Example: 80% of code is parallelizable (p=0.8):

    N=1:    Speedup = 1.0x
    N=2:    1/(0.2 + 0.4) = 1.67x
    N=4:    1/(0.2 + 0.2) = 2.5x
    N=8:    1/(0.2 + 0.1) = 3.33x
    N=16:   1/(0.2 + 0.05) = 4x
    N=∞:    1/0.2 = 5x     ← MAXIMUM SPEEDUP

    The serial portion (20%) limits maximum speedup to 5x
    regardless of how many cores you add!

    Practical Implication:
    "Optimization of the serial bottleneck gives more value
    than adding more hardware"

    Universal Scalability Law (USL) adds:
    - Coherency cost (contention): data synchronization overhead
    - N² scaling: caches, locks, coordination overhead
    → Throughput can DECREASE beyond a certain concurrency level!

# Amdahl's Law calculator
def amdahl_speedup(parallel_fraction: float, n_processors: int) -> float:
    """Calculate theoretical max speedup via Amdahl's Law."""
    serial = 1 - parallel_fraction
    return 1 / (serial + parallel_fraction / n_processors)

# Example: our service is 90% parallelizable
p = 0.90
for n in [1, 2, 4, 8, 16, 32, 64, 128]:
    s = amdahl_speedup(p, n)
    print(f"N={n:4d}: {s:.2f}x speedup")

# Output:
# N=   1: 1.00x
# N=   2: 1.82x
# N=   4: 3.08x
# N=   8: 4.71x
# N=  16: 6.40x
# N=  32: 7.80x
# N=  64: 8.77x
# N= 128: 9.35x
# Max (N=∞): 10x  (1/0.1 = 10)

26.4 Queueing Theory for Capacity Planning

    Little's Law (most important equation in capacity planning)
    ──────────────────────────────────────────────────────────

    L = λ × W

    L = Average number of items in the system (queue + service)
    λ = Average arrival rate (requests/second)
    W = Average time in system (latency)

    Example: API with 100 RPS, 200ms average latency:
    L = 100 × 0.2 = 20 concurrent requests in-flight at any time

    Usage 1: Calculate needed concurrency
    If latency target = 500ms and you need 100 RPS:
    Needed concurrency = 100 × 0.5 = 50 threads/goroutines

    Usage 2: Calculate max throughput
    If you have 50 threads and latency is 200ms:
    Max throughput = 50 / 0.2 = 250 RPS

    Utilization Law:
    ρ (utilization) = λ × service_time

    At ρ > 0.7: latency starts increasing
    At ρ > 0.9: latency explodes exponentially
    → Keep utilization < 70% for predictable latency

26.5 Capacity Planning Model

# Step 1: Measure current performance metrics
# For the past 30 days:
# - Max RPS (peak traffic)
# - CPU utilization at peak
# - Memory at peak
# - P99 latency at peak

# Step 2: Find saturation point
# Run load test, ramp up traffic, find where P99 starts degrading
# (This is your "saturation RPS")

# Step 3: Calculate headroom
# headroom_factor = saturation_rps / current_peak_rps
# headroom_factor should be > 2x (50% capacity used at peak)

# Step 4: Project future demand
# Using PromQL:
# predict_linear(sum(rate(http_requests_total[5m]))[30d:1h], 90*24*3600)
# This projects 90 days into the future based on 30-day trend

# Step 5: Calculate when you'll hit capacity
# days_until_saturation = log(saturation_rps / current_rps) / log(1 + daily_growth_rate)

python3 << 'EOF'
import math

current_rps = 1000
saturation_rps = 3000          # From load testing
daily_growth_rate = 0.005      # 0.5% daily growth ≈ 15%/month

days = math.log(saturation_rps / current_rps) / math.log(1 + daily_growth_rate)
print(f"Days until saturation: {days:.0f} days ({days/30:.1f} months)")
# Days until saturation: 220 days (7.3 months)
# → Need to add capacity before then!
EOF

26.6 Bottleneck Analysis

    The Theory of Constraints (Applied to Systems)
    ─────────────────────────────────────────────────

    In any system, ONE constraint limits total throughput.
    Improving anything that is NOT the bottleneck doesn't help.

    Steps:
    1. Identify the constraint (bottleneck)
    2. Exploit the constraint (maximize its throughput)
    3. Subordinate everything else (don't overwhelm non-constraints)
    4. Elevate the constraint (add capacity)
    5. Repeat (the next bottleneck will appear)

    How to Find the Bottleneck:
    ────────────────────────────
    Run load test → watch all resources
    First to hit 100% = bottleneck

    Resource order to check (typical web service):
    1. Application CPU (profile with perf/py-spy)
    2. Application threads (Little's Law: threads = RPS × latency)
    3. Database connections (connection pool exhaustion)
    4. Database CPU (pg_stat_activity)
    5. Database I/O (iostat for DB volume)
    6. Network bandwidth (sar -n DEV)
    7. Memory → swap → disk I/O cascade

26.7 Interview Questions

Q1: What is coordinated omission and why does it matter for benchmarking?

Answer: Coordinated omission happens when a benchmarking tool waits for the server to respond before sending the next request. When the server slows down, the tool slows its request rate — which means the benchmark is no longer measuring realistic load. In production, users don’t stop sending requests when the server is slow. Real-world slowdowns compound: if 100ms of requests pile up during a 1-second server pause, all 100ms of requests hit simultaneously when the server recovers. Tools like wrk2 avoid this by sending requests at a constant rate regardless of server speed, which gives accurate latency distributions (especially P99/P999).

Q2: Your service handles 1,000 RPS at 50ms average latency. How many concurrent requests are in-flight at any time?

Answer: Using Little’s Law: L = λ × W = 1,000 requests/sec × 0.050 sec = 50 concurrent requests in-flight at any time. This tells you: your thread pool, connection pool, or goroutine count must support at least 50 concurrent requests to handle this load. If you have only 20 threads and each takes 50ms, max throughput = 20/0.05 = 400 RPS — not enough. This is why thread pool sizing should always be calculated from Little’s Law: threads = RPS × latency_in_seconds.

26.8 Summary

Tool/Concept	Purpose
wrk2	Correct latency benchmarking (no coordinated omission)
k6	Scenario-based load testing
pgbench	PostgreSQL performance benchmarking
Amdahl’s Law	Maximum speedup from parallelization
Little’s Law	Concurrency = Throughput × Latency
Utilization Law	Keep resource utilization < 70%

Next Chapter: Chapter 27: eBPF, XDP & DPDK: Advanced Kernel Networking

Prerequisites

Chapter 23 (Performance Analysis).

Advantages & Disadvantages

Advantages: Proves system capacity before going live, validates tuning changes. Disadvantages: Synthetic benchmarks rarely match unpredictable real-world traffic patterns.

Common Mistakes

Benchmarking the page cache instead of the disk (not using O_DIRECT).
Running benchmarks for too short a time (e.g., 5 seconds) — misses GC pauses and thermal throttling.
Ignoring ‘coordinated omission’ — load generators stalling when the system under test stalls, underreporting tail latency.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Benchmark results vary wildly between runs	Background noise or thermal throttling	`Check dmesg for thermal warnings; monitor with top`	Ensure isolated environment; run for longer durations; check CPU governors
Storage benchmark shows impossible IOPS (>1M on SATA)	Benchmarking RAM/cache instead of disk	`Check fio config for direct=1`	Set direct=1 in fio to bypass page cache

Hands-on Lab

Objective: Benchmark CPU and Storage.

# 1. CPU Benchmark
sysbench cpu --cpu-max-prime=20000 --threads=4 run

# 2. Storage Benchmark (Random Read)
fio --name=randread --ioengine=libaio --iodepth=64 \
    --rw=randread --bs=4k --direct=1 --size=1G \
    --numjobs=1 --runtime=30 --time_based

Exercises

Run a fio benchmark to compare sequential write throughput against random write IOPS on your local disk.
Use wrk or hey to benchmark a local HTTP server. Observe how varying concurrency (-c) affects the P99 latency.

Revision Notes

Capacity planning determines when you will run out of resources based on current trends.
Always benchmark the bottleneck component in isolation if possible.
Focus on tail latency (P99, P99.9), not just average latency.
Use direct=1 (O_DIRECT) in fio to bypass the page cache for true storage testing.