CPU, Memory & Disk Performance Analysis

Chapter 23: CPU, Memory & Disk Performance Analysis

Learning Objectives

Systematically diagnose CPU, memory, and disk performance issues
Use the right tool for the right layer of the stack
Understand saturation, utilization, and errors for each resource
Apply the USE method in practice

What is this? (Beginner On-Ramp)

Performance analysis is the art of figuring out why a computer is slow. Instead of guessing, engineers use specific tools to check the ‘big four’ resources: CPU, Memory, Disk, and Network, finding exactly where the traffic jam is happening.

23.1 The Performance Analysis Toolset

    Linux Performance Tool Map
    ───────────────────────────

    Application Layer:
    strace  ltrace  gdb  valgrind

    System Call Interface:
    perf  eBPF/bpftrace

    CPU:
    top  htop  mpstat  vmstat  sar -u  perf stat  turbostat

    Memory:
    free  vmstat  /proc/meminfo  sar -r  numastat

    I/O (storage):
    iostat  iotop  pidstat  blktrace  biolatency

    Network:
    ss  ip  sar -n  netstat  nicstat  tcpdump  iperf3

    Kernel:
    dmesg  /proc/*  /sys/*  slabtop  sysctl

23.2 CPU Performance Analysis

# ── CPU Utilization (USE Method) ──────────────────────────
# Utilization:
mpstat -P ALL 1                    # Per-CPU utilization every 1s
# Key columns: %usr %sys %iowait %steal %idle

# Reading mpstat output:
# %usr:    User-space time (your apps)
# %sys:    Kernel time (syscalls, interrupts)
# %iowait: Waiting for I/O (disk/network bottleneck signal)
# %steal:  VM steal (hypervisor taking CPU → cloud issue)
# %idle:   Free

# Saturation:
vmstat 1
# r column: run queue — number of processes WAITING for CPU
# If r > nproc (number of CPUs) = CPU saturated

uptime
# load average: 1, 5, 15 minutes
# load > nproc = CPU saturated

# ── Per-Process CPU ───────────────────────────────────────
top               # Dynamic process list (sort by CPU: press P)
htop              # Better top
pidstat -u 1      # Per-process CPU utilization (per second)

# Which CPU is a process running on?
ps -o pid,psr,comm -p $(pidof nginx)
# psr = processor (CPU core) it's running on

# ── CPU Frequency and Throttling ──────────────────────────
cat /proc/cpuinfo | grep MHz
turbostat --interval 5 2>/dev/null    # Intel: detailed freq info

# Check if CPU is being throttled (thermal/power)
dmesg | grep -i "throttl\|thermal\|cpufreq"

# ── Context Switches ──────────────────────────────────────
vmstat 1 | awk '{print $12, $13}'   # cs (context switches), in (interrupts)
pidstat -w 1                         # Per-process context switches

# High voluntary CS: processes waiting for I/O (expected)
# High involuntary CS: too many processes sharing CPUs (scheduling pressure)

23.3 Memory Performance Analysis

# ── Memory Overview ───────────────────────────────────────
free -h
# Total: Total RAM
# Used: Used by processes + kernel
# Free: Truly free (rare to be large; Linux uses RAM for cache)
# Available: What's available to start new apps (free + reclaimable cache)
# Buff/cache: Page cache (can be reclaimed)
# Swap: If swap used → memory pressure

cat /proc/meminfo
# MemTotal, MemFree, MemAvailable
# Buffers (block device cache), Cached (file cache), SwapCached
# Active/Inactive: Recently/not-recently accessed pages
# Dirty: Pages waiting to be written to disk
# HugePages_Total, HugePages_Free

# ── Memory Saturation ─────────────────────────────────────
# Swap usage (saturation signal):
vmstat 1 | awk '{print $7, $8}'   # si (swap in), so (swap out)
# si/so > 0 = swapping occurring = memory saturated

# Swap in/out per process:
sar -W 1                          # System-wide swap rate

# ── Page Faults ───────────────────────────────────────────
# Minor fault: page not in TLB but in RAM (cheap)
# Major fault: page not in RAM, must read from disk (expensive!)
pidstat -f 1                      # Per-process page faults

# Watch for major page faults:
while true; do
    awk '/pgmajfault/ {print "Major page faults/s:", $2}' /proc/vmstat
    sleep 1
done

# ── Memory per Process ────────────────────────────────────
# RSS (Resident Set Size): Physical RAM used
# VSZ (Virtual Size): Virtual address space (includes mapped files)
ps aux --sort=-%mem | head -10

# Detailed memory map:
cat /proc/$(pidof postgres)/smaps_rollup
# Rss, Pss, Private_Dirty, Shared_Clean

# ── OOM Analysis ──────────────────────────────────────────
dmesg -T | grep -A5 "Out of memory"
# Shows: which process was killed, its oom_score, memory state at kill

23.4 Disk I/O Performance Analysis

# ── iostat: The Primary Disk Tool ────────────────────────
iostat -xz 1              # Extended stats, skip idle devices, 1s interval

# Key columns:
# r/s:    reads per second
# w/s:    writes per second
# rkB/s:  read throughput (KB/s)
# wkB/s:  write throughput (KB/s)
# r_await: read latency (ms) — average wait time per read
# w_await: write latency (ms) — average wait time per write
# %util:   how busy the device is (100% = saturated)
# aqu-sz:  average queue depth (>1 = queuing = saturated)

# ── USE for Disk ──────────────────────────────────────────
# Utilization: %util column in iostat
# Saturation:  aqu-sz > 1 OR await is high
# Errors:      dmesg | grep "I/O error"

# Latency thresholds:
# HDD: reads < 10ms OK, > 20ms concerning
# SATA SSD: reads < 1ms OK, > 5ms concerning
# NVMe: reads < 0.1ms OK, > 1ms concerning

# ── Per-Process I/O ───────────────────────────────────────
iotop -o              # Show only processes with I/O (-o = only active)
pidstat -d 1          # Per-process disk I/O stats

# ── Block-Level Analysis ──────────────────────────────────
# Which files are being read/written?
opensnoop               # bcc tool: trace all open() calls
# Or:
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s: %s\n", comm, str(args->filename)); }'

# ── Disk Errors ───────────────────────────────────────────
dmesg | grep -i "error\|failed\|reset\|timeout" | grep -i "sd\|nvme\|ata"
smartctl -a /dev/sda    # SMART disk health data

23.5 The 60-Second Checklist

#!/bin/bash
# 60-second-triage.sh — Quick performance triage

echo "=== 1. LOAD & UPTIME ==="
uptime

echo "=== 2. TOP PROCESSES (CPU) ==="
ps aux --sort=-%cpu | head -6

echo "=== 3. MEMORY ==="
free -h

echo "=== 4. VMSTAT (1 second) ==="
vmstat 1 5

echo "=== 5. CPU PER-CORE ==="
mpstat -P ALL 1 1

echo "=== 6. DISK I/O ==="
iostat -xz 1 3

echo "=== 7. NETWORK ==="
sar -n DEV 1 3

echo "=== 8. TOP DISK USERS ==="
iotop -o -b -n 1 | head -10

echo "=== 9. KERNEL MESSAGES ==="
dmesg -T | tail -10

echo "=== 10. SWAP ==="
vmstat -s | grep -E "swap|page"

23.6 Production Scenarios

Scenario: High iowait, application slow

# iowait > 30% in mpstat output

# Step 1: Which disk?
iostat -xz 1 | grep -v "^$"
# Find device with %util near 100% or high await

# Step 2: Which process?
iotop -o
# Look for process consuming most disk I/O

# Step 3: What files?
lsof -p $(pidof slow-process)
# See which files are open

# Step 4: What type of I/O?
iostat -x 1 | grep -A2 "Device"
# High r_await = reads are slow (cache misses, random access)
# High w_await = writes backing up (sync writes, small writes)

# Step 5: Latency histogram
biolatency -D 10    # BCC tool: latency by device

23.7 Interview Questions

Q1: What does high %iowait in mpstat output mean?

Answer: %iowait is the percentage of time the CPU was idle because a process was waiting for I/O (typically disk or network). High iowait means the CPU has work to do but can’t proceed because it’s waiting on I/O. Common causes: (1) Slow disk (HDD, degraded RAID). (2) Insufficient disk bandwidth — too much I/O queuing up. (3) Database doing full table scans (reading large amounts from disk). (4) Swapping (memory pressure forcing reads from swap). Important: iowait is an idle state — CPUs aren’t busy, but they CAN’T be busy. First find which device is slow (iostat -x), then which process (iotop), then which files (lsof).

Q2: What is the difference between RSS and VSZ in process memory?

Answer: VSZ (Virtual Size) is the total virtual address space of the process — includes code, stack, heap, memory-mapped files, shared libraries, and reserved-but-unused memory. It can be much larger than physical RAM without that being a problem. RSS (Resident Set Size) is the actual physical RAM pages the process is currently using. RSS is what actually matters for memory pressure. Caveats: RSS double-counts shared pages (two processes sharing a 10MB library each show 10MB RSS, but only 10MB physical RAM is used). PSS (Proportional Set Size) is more accurate — divides shared pages proportionally. View in /proc/<pid>/smaps_rollup.

23.8 Summary

Resource	Utilization Metric	Saturation Metric	Tool
CPU	`mpstat %usr+%sys`	`vmstat r > nproc`	`top`, `mpstat`
Memory	`free MemAvailable`	`vmstat si/so > 0`	`free`, `vmstat`
Disk	`iostat %util`	`iostat aqu-sz > 1`	`iostat`, `iotop`
Network	`sar -n DEV rxkB/s`	Dropped packets	`sar`, `ss`

Next Chapter: Chapter 24: perf, eBPF & Flamegraphs

Prerequisites

Chapter 3 (Process), Chapter 4 (Memory).

Advantages & Disadvantages

Advantages: Replaces guessing with data-driven fixes, identifies the exact bottleneck (CPU, Disk, RAM, Net). Disadvantages: Requires deep system knowledge to interpret the metrics correctly.

Common Mistakes

Assuming high CPU usage is always bad — 100% CPU on a batch processing job means it’s working efficiently.
Looking at average CPU instead of per-core — 100% on one core out of 16 means 6% average, masking a single-thread bottleneck.
Ignoring I/O wait (wa) — high wa means the CPU is idling waiting for disks.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
High load average, low CPU usage	Processes stuck in D state (I/O wait)	`vmstat 1; iostat -x 1`	Fix disk bottleneck or network storage issue
High system (sy) CPU	High syscall rate, context switching	`perf stat -e context-switches; strace -c`	Optimize app to batch I/O, fix lock contention

Hands-on Lab

Objective: Perform basic performance profiling.

# 1. 60-second dashboard
uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
top

Exercises

Use stress-ng to simulate CPU, memory, and I/O load. Use top, vmstat, and iostat to identify which subsystem is stressed in each test.
Find the process doing the most context switches using pidstat -w 1.

Revision Notes

USE Method: Utilization, Saturation, Errors for every resource.
Load average: Exponential moving average of runnable and uninterruptible (D state) threads.
Always check dmesg first for kernel errors or hardware issues.
Distinguish between user time (us) and system time (sy).