Capacity Planning & Performance Benchmarking

Chapter 44: Capacity Planning & Performance Benchmarking

Learning Objectives

Understand capacity planning methodology for production systems
Build demand forecasting models using historical metrics
Benchmark systems properly with load testing tools
Identify and resolve performance bottlenecks systematically

What is this? (Beginner On-Ramp)

Capacity planning is the math used to predict the future. By analyzing how much CPU and disk space an application uses today, engineers calculate exactly when they need to buy more servers before the system runs out of resources and crashes.

31.1 Capacity Planning Framework

    Capacity Planning Process
    ──────────────────────────

    1. MEASURE: What does the system do now?
       • Current RPS, latency, resource utilization
       • Saturation points (when does it start degrading?)

    2. FORECAST: What will demand look like in 6-12 months?
       • Historical growth trends
       • Planned features, marketing events, seasonal patterns

    3. PROVISION: How much capacity do we need?
       • Target utilization (60-70% for headroom)
       • Safety buffer for traffic spikes (2x peak capacity)

    4. OPTIMIZE: Are we using capacity efficiently?
       • Right-sizing (not over-provisioned)
       • Caching, efficiency improvements

    The Goal:
    ──────────
    Enough capacity to serve peak demand + safety headroom
    Not so much that you're wasting money on idle resources

31.2 Demand Forecasting

    Growth Modeling
    ────────────────

    Approach 1: Linear Growth
    Future demand = Current demand × (1 + growth_rate)^months

    Example: 1000 RPS today, 15% monthly growth:
    Month 6:  1000 × 1.15^6  = 2,313 RPS
    Month 12: 1000 × 1.15^12 = 5,350 RPS

    Approach 2: Seasonal Adjustment
    E-commerce: 3x spike during Black Friday
    Tax software: 5x spike in April
    Plan for: baseline + maximum seasonal spike

    Approach 3: Event-driven
    Marketing campaign scheduled → +200% traffic expected
    Plan capacity before the campaign, not during

# PromQL: Predict future RPS from 30-day trend
# Linear regression over 30 days projected 90 days forward
predict_linear(
  sum(rate(http_requests_total[1h]))[30d],
  90 * 24 * 3600
)

# Actual growth rate (week-over-week)
sum(rate(http_requests_total[5m]))
  /
sum(rate(http_requests_total[5m] offset 7d))
- 1

31.3 Resource Utilization Targets

    Target Utilization Guidelines
    ───────────────────────────────

    CPU:
    ✓ Target 60-70% sustained utilization
    ✓ Leave 30-40% headroom for:
      - Traffic spikes (sudden 2x)
      - GC pauses (JVM, Go)
      - Background jobs
      - System overhead
    ✗ 90%+ sustained = at risk, add capacity now

    Memory:
    ✓ Target 70-80% utilization
    ✓ Leave 20-30% for:
      - Burst allocations
      - Cache warming after restarts
      - Buffer cache
    ✗ 90%+ = OOM risk, scale up

    Disk I/O:
    ✓ Target < 60% utilization
    ✓ High queue depth = saturated
    ✓ Check await time (> 100ms for HDD, > 10ms for SSD = saturated)

    Network:
    ✓ Target < 50% of NIC capacity
    ✓ 1Gbps NIC: target < 500Mbps sustained

    Database Connections:
    ✓ Target < 70% of max_connections
    ✓ Connection pooler (pgBouncer) at 80% of its pool

31.4 Load Testing with k6

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

// Custom metrics
const errorRate = new Rate('errors');
const checkoutLatency = new Trend('checkout_latency');

// Test stages (ramp up, sustain, ramp down)
export const options = {
  stages: [
    { duration: '2m', target: 100 },   // Ramp up to 100 VUs over 2 minutes
    { duration: '5m', target: 100 },   // Stay at 100 VUs for 5 minutes
    { duration: '2m', target: 500 },   // Ramp up to 500 VUs
    { duration: '5m', target: 500 },   // Stay at 500 VUs
    { duration: '2m', target: 0 },     // Ramp down to 0
  ],
  thresholds: {
    // SLO validation: fail test if SLO not met
    http_req_duration: ['p(99)<200'],       // P99 < 200ms
    errors: ['rate<0.01'],                   // Error rate < 1%
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  // Checkout flow
  const startTime = Date.now();

  const cartResponse = http.post(
    'https://api.example.com/cart',
    JSON.stringify({ product_id: 'SKU-123', quantity: 1 }),
    { headers: { 'Content-Type': 'application/json' } }
  );

  check(cartResponse, {
    'cart created': (r) => r.status === 201,
  });

  const checkoutResponse = http.post(
    'https://api.example.com/checkout',
    JSON.stringify({
      cart_id: cartResponse.json('cart_id'),
      payment: { card: '4242424242424242', exp: '12/26', cvv: '123' },
    }),
    { headers: { 'Content-Type': 'application/json' } }
  );

  const success = check(checkoutResponse, {
    'checkout success': (r) => r.status === 200,
    'latency OK': (r) => r.timings.duration < 200,
  });

  errorRate.add(!success);
  checkoutLatency.add(Date.now() - startTime);

  sleep(1);
}

# Run k6 test
k6 run load-test.js

# Run with output to InfluxDB (for Grafana)
k6 run --out influxdb=http://influxdb:8086/k6 load-test.js

# Key metrics to watch:
# http_req_duration: Request latency (P50, P90, P95, P99)
# http_req_failed: % of failed requests
# iterations: Total test iterations
# vus: Active virtual users
# http_reqs: Total requests made

31.5 Benchmarking Tools

# ── HTTP Benchmarking ─────────────────────────────────────
# wrk: Simple high-performance benchmark
wrk -t12 -c400 -d30s --latency http://api.example.com/health
# -t: threads, -c: connections, -d: duration
# Output: Requests/sec, Latency P50/P75/P90/99, Errors

# ab (ApacheBench): Quick single-endpoint test
ab -n 10000 -c 100 http://api.example.com/health
# -n: total requests, -c: concurrent requests

# hey: Go-based benchmark tool
hey -n 100000 -c 200 -q 1000 http://api.example.com/checkout
# -q: rate limit (queries per second)

# ── Database Benchmarking ─────────────────────────────────
# pgbench: PostgreSQL benchmarking
pgbench -i -s 100 mydb                    # Initialize (100x scale factor)
pgbench -c 50 -j 10 -T 60 mydb           # 50 clients, 10 threads, 60 seconds
# Output: TPS (transactions per second), latency

# sysbench: MySQL/PostgreSQL OLTP benchmark
sysbench oltp_read_write \
    --mysql-host=localhost \
    --mysql-port=3306 \
    --mysql-user=root \
    --tables=10 \
    --table-size=1000000 \
    prepare

sysbench oltp_read_write \
    --threads=16 \
    --time=300 \
    run

# ── Disk Benchmarking ─────────────────────────────────────
# fio: Flexible I/O tester
# Random 4K read (IOPS-focused, database-like)
fio --name=rand-read --ioengine=libaio --iodepth=64 \
    --rw=randread --bs=4k --numjobs=4 \
    --size=10G --filename=/dev/nvme0n1 \
    --runtime=60 --time_based \
    --output-format=json

# Sequential read (throughput-focused, streaming)
fio --name=seq-read --ioengine=libaio --iodepth=32 \
    --rw=read --bs=128k --numjobs=1 \
    --size=10G --filename=/dev/nvme0n1 \
    --runtime=60 --time_based

# ── Memory Bandwidth ─────────────────────────────────────
# stream: Memory bandwidth benchmark
./stream
# Output: Copy, Scale, Add, Triad bandwidth in MB/s

31.6 Finding the Bottleneck

    Bottleneck Identification Process
    ────────────────────────────────────

    Method: Saturate one resource at a time

    1. Run load test → watch all resources simultaneously
    2. First resource to reach 100% utilization = bottleneck
    3. Fix that bottleneck (add capacity, optimize, cache)
    4. Repeat from step 1

    Common Bottleneck Hierarchy (web services):
    ────────────────────────────────────────────
    1. CPU: Application code hotspot
       Fix: Profile → optimize hot path, add caching

    2. Network (bandwidth): Large responses
       Fix: Compression, CDN, response size reduction

    3. Disk I/O (database): Too many queries, no index
       Fix: Query optimization, indexes, read replicas, caching

    4. Memory: Working set doesn't fit → swap → I/O
       Fix: Increase RAM, tune page cache, add caching layer

    5. Database connections: Connection pool exhaustion
       Fix: PgBouncer, connection pooling, reduce queries

    6. Network (latency): Many small requests
       Fix: Connection reuse (keep-alive), batching, local caching

    Amdahl's Law:
    ─────────────
    Maximum speedup from optimizing X% of code:
    Speedup = 1 / (1 - X/100 + X/100/N)

    If 50% of time is in one function:
    Optimize that function → max 2x speedup, not infinity!
    The other 50% becomes the new bottleneck.

31.7 Interview Questions

Q1: How do you approach capacity planning for a service that handles 10,000 RPS today and is expected to grow 20% month-over-month?

Answer: (1) Baseline: Measure current resource utilization at 10K RPS (CPU, memory, disk, network). Find saturation points with load testing — what RPS causes >70% CPU? (2) Project demand: 10K × 1.20^12 = ~74K RPS in 12 months. Plan for peak traffic (usually 2-3x average, so ~150K RPS for planning). (3) Scale calculation: If one server handles 2K RPS sustainably, need 75 servers for 150K RPS. Add 20% headroom → 90 servers. (4) Cost optimize: Before adding hardware, check if caching (Redis) can reduce RPS hitting origin, or if code optimization can increase capacity per server. (5) Trigger points: Set up alerts at 70% capacity to trigger provisioning before hitting the wall.

Q2: What is the difference between a load test and a stress test?

Answer: A load test simulates expected production traffic levels to verify performance SLOs are met under normal and peak load. It answers: “Can our service handle X RPS with P99 < 200ms?” A stress test pushes the system beyond its expected limits to find the breaking point. It answers: “At what point does our service start degrading? What breaks first? How does it fail?” Stress tests are important for understanding your headroom and knowing what failure looks like before it happens in production. A third type: spike test — sudden extreme traffic burst to test elasticity (auto-scaling, connection pool behavior).

31.8 Summary

Topic	Key Point
Utilization targets	CPU 60-70%, Memory 70-80%
Load testing	k6, wrk, hey for HTTP; pgbench for DB
Bottleneck finding	Saturate one resource at a time
Forecasting	PromQL predict_linear, historical trends
Amdahl’s Law	Optimizing 50% → max 2x speedup

Next Chapter: Chapter 32: Container Security

Prerequisites

Chapter 23 (Performance Analysis).

Advantages & Disadvantages

Advantages: Prevents “hug of death” outages, optimizes cloud spend. Disadvantages: Difficult to predict viral traffic spikes, mathematical models often fail reality.

Common Mistakes

Assuming linear scaling (e.g., 2x servers = 2x throughput) — ignores database bottlenecks and lock contention.
Planning based on average load instead of peak load.
Not factoring in the time required to procure hardware or scale cloud limits.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
System crashes during peak event	Capacity planned for average load	`Review monitoring data during crash`	Plan for peak traffic (e.g., Black Friday) + safety margin
Cloud bill unexpectedly high	Over-provisioning without auto-scaling	`Review cloud billing dashboard`	Implement horizontal pod autoscaling (HPA) or cloud auto-scaling groups

Hands-on Lab

Objective: Extrapolate resource usage.

# Conceptual:
# If disk usage is growing at 5GB/day and you have 100GB free, when will you run out?
# Calculate: 100 / 5 = 20 days.
# Action: Set an alert to trigger at 14 days remaining.

Exercises

Review the CPU usage of a service over the last 30 days. Use linear regression (conceptually or via a spreadsheet) to predict when it will hit 80% utilization.
Explain the difference between vertical scaling (scaling up) and horizontal scaling (scaling out). When is each appropriate?

Revision Notes

Capacity planning ensures you have enough resources to handle future demand before it becomes an emergency.
Requires accurate historical metrics (Observability).
Auto-scaling helps handle dynamic load, but limits and quotas must still be planned for.
Load testing is critical to validate capacity models.