CPU Affinity, IRQ Balancing & Storage Tuning

Chapter 13: CPU Affinity, IRQ Balancing & Storage Tuning

Learning Objectives

Understand CPU affinity and when to pin processes to CPUs
Understand IRQ balancing and network interrupt distribution
Master storage tuning for SSDs and HDDs

What is this? (Beginner On-Ramp)

CPU and Storage tuning is about maximizing hardware efficiency. It involves telling the system exactly which CPU cores should handle which tasks (pinning) and configuring how disk requests are organized (I/O scheduling) to prevent fast SSDs from waiting on slow software.

13.1 CPU Affinity

CPU affinity pins a process or thread to specific CPU cores, preventing the scheduler from moving it to other cores.

    Why CPU Affinity Matters
    ─────────────────────────

    Without CPU affinity:
    Process A runs on CPU 0 → Scheduler migrates to CPU 4
    → CPU 0's L1/L2 cache data for Process A becomes stale
    → Process A on CPU 4 has cache miss on first memory access
    → Cache warm-up takes time = latency spike

    With CPU affinity (pinned to CPU 0):
    Process A always on CPU 0
    → L1/L2 cache always warm for Process A
    → Consistent low-latency performance
    → NUMA-aware (process stays on same socket's RAM)

# ── View CPU Affinity ─────────────────────────────────────
taskset -p <PID>                   # Show CPU affinity mask
# cpu affinity mask: ff            (hex ff = CPUs 0-7)

# ── Set CPU Affinity for New Process ─────────────────────
taskset -c 0,1,2,3 nginx           # Run nginx on CPUs 0-3

# ── Change Affinity of Running Process ────────────────────
taskset -cp 0,1 <PID>             # Pin PID to CPUs 0 and 1
taskset -p 0x3 <PID>              # Same, using hex mask (0x3 = CPUs 0-1)

# ── Thread-Level Affinity (using pthread) ─────────────────
# In C code:
# cpu_set_t cpuset;
# CPU_SET(cpu_id, &cpuset);
# pthread_setaffinity_np(thread, sizeof(cpuset), &cpuset);

# ── CPU Isolation (isolcpus) ──────────────────────────────
# Reserve CPUs exclusively for specific processes
# Add to GRUB kernel parameters:
# isolcpus=2,3,4,5 nohz_full=2,3,4,5 rcu_nocbs=2,3,4,5

# After boot, only explicitly assigned processes run on CPUs 2-5
# All kernel threads and normal processes use CPUs 0,1

# Verify isolation:
cat /sys/devices/system/cpu/isolated

# Assign a high-priority process to isolated CPUs:
taskset -c 2,3,4,5 chrt -f 99 ./realtime_app

CPU Isolation for Low-Latency Applications

# GRUB_CMDLINE_LINUX configuration for a trading/RT system
GRUB_CMDLINE_LINUX="isolcpus=2-15 nohz_full=2-15 rcu_nocbs=2-15 \
    irqaffinity=0,1 kthread_cpus=0,1"

# This configuration:
# - Reserves CPUs 2-15 for application use only
# - CPUs 0,1 handle all kernel work, IRQs, system threads
# - Dramatically reduces jitter on isolated CPUs

# Verify with cyclictest (real-time latency test):
cyclictest -t 4 -p 80 -n -l 1000000 --cpu=2,3,4,5
# Should see max latency < 100μs on isolated CPUs

13.2 IRQ Balancing

IRQs (Interrupt Requests) are signals from hardware to the CPU when a device needs attention (network packet received, disk I/O complete). Distributing IRQs across CPUs improves performance.

    IRQ Distribution Problem
    ─────────────────────────

    Without IRQ balancing:
    All network packet interrupts → CPU 0
    CPU 0 is 100% busy handling interrupts
    CPU 1-15 idle
    → Single CPU bottleneck for high-throughput networks

    With IRQ balancing (irqbalance or manual):
    CPU 0: handles NIC queue 0 interrupts
    CPU 1: handles NIC queue 1 interrupts
    CPU 2: handles NIC queue 2 interrupts
    ...
    → Each CPU handles its own NIC queue
    → 16x more network interrupt throughput

# ── View IRQ Statistics ───────────────────────────────────
cat /proc/interrupts
# Column headers: CPU0 CPU1 CPU2 ... then IRQ description
# Look at which CPUs are handling which IRQs

# Find IRQ number for your NIC
grep eth0 /proc/interrupts

# ── irqbalance Daemon ────────────────────────────────────
# Automatically distributes IRQs across CPUs
systemctl enable --now irqbalance
systemctl status irqbalance

# irqbalance configuration
cat /etc/irqbalance.conf
# IRQBALANCE_BANNED_CPUS=0,1   # Don't put IRQs on CPUs 0,1 (for isolcpus)

# ── Manual IRQ Affinity ───────────────────────────────────
# View current IRQ affinity (which CPUs handle which IRQ)
cat /proc/irq/<IRQ_NUMBER>/smp_affinity
# f = CPUs 0-3 (all of them for 4-CPU system)
# 1 = CPU 0 only
# 2 = CPU 1 only
# 3 = CPUs 0 and 1

# Set IRQ affinity for NIC (multi-queue setup)
# Get IRQ numbers for eth0 queues
grep eth0 /proc/interrupts | awk '{print $1}' | tr -d ':'

# Assign NIC queue IRQs to different CPUs
NIC_IRQS=($(grep eth0 /proc/interrupts | awk '{print $1}' | tr -d ':'))
for i in "${!NIC_IRQS[@]}"; do
    CPU=$((i % $(nproc)))
    echo $((1 << CPU)) > /proc/irq/${NIC_IRQS[$i]}/smp_affinity
    echo "IRQ ${NIC_IRQS[$i]} → CPU $CPU"
done

# ── Set RSS (Receive Side Scaling) ────────────────────────
# Modern NICs can steer packets to specific queues based on TCP 5-tuple
# This ensures a TCP flow is always handled by the same CPU
# (good for cache affinity within a connection)

# Enable RSS:
ethtool -X eth0 equal $(nproc)    # Distribute equally across CPUs

# Verify queue distribution:
for i in $(seq 0 $(( $(nproc) - 1 ))); do
    echo -n "CPU $i: "
    cat /sys/class/net/eth0/queues/rx-$i/rps_cpus 2>/dev/null || echo "N/A"
done

13.3 Storage Tuning

I/O Schedulers

    I/O Scheduler Comparison
    ─────────────────────────

    none:
    - No scheduling — requests go directly to driver
    - Best for SSDs/NVMe (hardware already does reordering)
    - Lowest latency for fast storage

    mq-deadline:
    - Deadline-based scheduling
    - Prevents starvation (old requests get priority)
    - Best for HDDs and mixed workloads
    - Default for most distros on SATA disks

    kyber:
    - Targets specific latency goals
    - Good for SSDs with multiple workloads
    - Designed for high-speed storage

    bfq:
    - Budget Fair Queueing
    - Optimal for multiple competing processes
    - Higher latency, better fairness
    - Best for desktop, not servers
    - BAD for NVMe (adds unnecessary latency)

# ── View and Change I/O Scheduler ─────────────────────────
# View scheduler for a device
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq   ← [none] is active

cat /sys/block/sda/queue/scheduler
# mq-deadline [bfq] kyber none   ← [bfq] is active (bad for servers!)

# Change scheduler
echo "none" > /sys/block/nvme0n1/queue/scheduler          # NVMe SSD
echo "mq-deadline" > /sys/block/sda/queue/scheduler       # SATA HDD/SSD

# ── Persistent via udev rule ───────────────────────────────
cat > /etc/udev/rules.d/60-io-scheduler.rules << 'EOF'
# NVMe SSDs: use none (no scheduling overhead)
ACTION=="add|change", KERNEL=="nvme*", ATTR{queue/scheduler}="none"
# SATA SSDs: use none or mq-deadline
ACTION=="add|change", KERNEL=="sd*", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"
# HDDs: use mq-deadline
ACTION=="add|change", KERNEL=="sd*", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"
EOF

udevadm control --reload-rules
udevadm trigger

Block Device Queue Tuning

# ── Queue Depth ───────────────────────────────────────────
# Higher queue depth = more I/O in flight = higher throughput
# but also higher latency
cat /sys/block/nvme0n1/queue/nr_requests
# Default: 64-256

# Increase for high-throughput workloads
echo 512 > /sys/block/nvme0n1/queue/nr_requests

# ── Read Ahead ────────────────────────────────────────────
# How many KB to prefetch on sequential reads
# Default: 128KB (often too low for sequential workloads)
blockdev --getra /dev/nvme0n1           # Current (in 512-byte sectors)
blockdev --setra 4096 /dev/nvme0n1     # Set to 2MB (4096 × 512)

# Disable for random I/O workloads (DB):
blockdev --setra 0 /dev/nvme0n1

# ── Write Cache ───────────────────────────────────────────
hdparm -I /dev/sda | grep "Write cache"   # Check if enabled
hdparm -W 1 /dev/sda                       # Enable write cache
# WARNING: enables write cache but data can be lost on power failure
# Use only with a UPS or battery-backed RAID controller

# For databases: disable write cache to ensure fsync correctness
hdparm -W 0 /dev/sda

LVM Performance Tuning

# ── LVM Configuration ─────────────────────────────────────
# Create LVM with striping (RAID 0) for performance
pvcreate /dev/nvme0n1 /dev/nvme1n1
vgcreate vg_data /dev/nvme0n1 /dev/nvme1n1

# Create striped LV (stripes across 2 PVs, 256KB stripe size)
lvcreate -l 100%FREE -i 2 -I 256 -n lv_data vg_data

# ── Monitor I/O per LV ────────────────────────────────────
dmsetup status /dev/vg_data/lv_data
lvdisplay --maps /dev/vg_data/lv_data

NVMe Tuning

# ── NVMe Queue Settings ───────────────────────────────────
# Modern NVMe SSDs support multiple queues (multi-queue)
ls /sys/block/nvme0n1/mq/
# 0  1  2  3  4  5  6  7  ... (one directory per queue)

# Each queue can be assigned to a CPU core
# This is done automatically by the kernel

# ── NVMe Power Management ─────────────────────────────────
# Disable power saving for performance
echo "performance" > /sys/block/nvme0n1/device/power/pm_qos_latency_tolerance_us

# ── Enable Write Merging ──────────────────────────────────
cat /sys/block/nvme0n1/queue/nomerges
echo 0 > /sys/block/nvme0n1/queue/nomerges   # Enable merging

# ── Verify NVMe Performance ───────────────────────────────
# Sequential read test:
fio --name=seqread --ioengine=libaio --iodepth=32 \
    --rw=read --bs=128k --numjobs=1 \
    --size=10G --filename=/dev/nvme0n1 \
    --output-format=json

# Random 4K read (IOPS test):
fio --name=randread --ioengine=libaio --iodepth=64 \
    --rw=randread --bs=4k --numjobs=4 \
    --size=10G --filename=/dev/nvme0n1 \
    --output-format=json

13.4 CPU Frequency Scaling

# ── CPU Governor ──────────────────────────────────────────
# Governor controls how the kernel manages CPU frequency
# performance: always maximum frequency
# powersave:   always minimum frequency
# ondemand:    scale with load (has ramp-up latency)
# schedutil:   modern, scales based on scheduler hints

# View current governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Set all CPUs to performance governor
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo "performance" > $cpu
done

# Permanent (using tuned):
tuned-adm profile throughput-performance    # Server workloads
tuned-adm profile latency-performance       # Low-latency

# ── Check CPU Frequency ───────────────────────────────────
cat /proc/cpuinfo | grep MHz
# Or:
turbostat --interval 5                      # Intel detailed frequency

13.5 Production Configuration Template

#!/bin/bash
# production-tuning.sh - Apply all performance tuning

set -euo pipefail

echo "=== Applying Production Performance Tuning ==="

# I/O Scheduler
for dev in /sys/block/nvme*/queue/scheduler; do
    echo "none" > "$dev"
done
for dev in /sys/block/sd*/queue/scheduler; do
    rotational=$(cat "$(dirname $dev)/rotational")
    if [ "$rotational" = "0" ]; then
        echo "none" > "$dev"
    else
        echo "mq-deadline" > "$dev"
    fi
done
echo "✓ I/O schedulers configured"

# CPU Governor
for gov in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo "performance" > "$gov" 2>/dev/null || true
done
echo "✓ CPU governor set to performance"

# Disable THP
echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
echo "✓ THP disabled"

# IRQ Balance
systemctl enable --now irqbalance
echo "✓ IRQ balancing enabled"

echo "=== Production tuning applied ==="

13.6 Interview Questions

Q1: When would you use CPU affinity (pinning) in production?

Answer: CPU affinity is beneficial when: (1) Low-latency applications: trading systems, real-time data processing — pinning eliminates jitter from CPU migration. (2) Cache-hot workloads: if a process has a large working set and benefits from cache warmth, pinning prevents cache invalidation from migrations. (3) NUMA optimization: pinning to CPUs that are local to the process’s memory prevents cross-NUMA accesses. (4) Competing workloads: isolate a CPU-hungry process from other processes. Downsides: reduces scheduler flexibility, doesn’t automatically adapt to load changes. For most general-purpose services, letting the scheduler decide is better.

Q2: What is the difference between IRQ affinity and RSS?

Answer: IRQ affinity determines which CPU core handles the hardware interrupt when the NIC receives a packet. RSS (Receive Side Scaling) is a NIC feature that distributes incoming packets across multiple NIC queues based on a hash of the packet’s 5-tuple (src/dst IP, src/dst port, protocol), so packets from the same TCP flow always go to the same queue and are processed by the same CPU. Together: RSS assigns flows to queues, IRQ affinity assigns queue interrupts to CPUs. The combination ensures that a TCP flow’s packets are consistently processed by one CPU, maximizing cache efficiency.

13.7 Summary

Technique	Use Case	Command
CPU affinity	Pin process to CPUs	`taskset -c 0,1 ./app`
IRQ balancing	Distribute interrupts	`irqbalance` daemon
I/O scheduler none	NVMe SSDs	`echo none > .../scheduler`
I/O scheduler mq-deadline	HDDs	`echo mq-deadline > .../scheduler`
Performance governor	Production servers	`echo performance > .../scaling_governor`
Disable THP	All servers	`echo never > .../enabled`

Next Chapter: Chapter 14: Linux Security Fundamentals

Prerequisites

Chapter 10 (sysctl), Chapter 3 (Process Lifecycle) — understand CPU scheduling and block I/O.

Advantages & Disadvantages

Advantages: CPU pinning eliminates cache misses, I/O scheduling maximizes SSD throughput. Disadvantages: Over-tuning reduces the kernel’s ability to balance dynamic workloads naturally.

Common Mistakes

Pinning all application threads to one CPU — CPU affinity is powerful but creates hot spots if too many threads share one core.
Assuming SSDs need the same I/O scheduler as HDDs — SSDs use none or mq-deadline; HDDs use bfq or kyber.
Using irqbalance alongside manual IRQ affinity — they conflict; disable irqbalance if you set affinity manually.
Not accounting for NUMA in CPU pinning — pinning CPU to node 0 but memory on node 1 wastes the benefit.
Setting transparent_hugepage=always for mixed workloads — THP benefits NUMA-aware apps but harms databases; use per-workload tuning.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Specific CPU core at 100%, others idle	IRQ affinity or process affinity too narrow	`mpstat -P ALL 1; cat /proc/interrupts	grep eth0`
High I/O latency on NVMe	Wrong I/O scheduler (using cfq/bfq on NVMe)	`cat /sys/block/nvme0n1/queue/scheduler`	echo none > /sys/block/nvme0n1/queue/scheduler
Database I/O inconsistency	I/O priority not set — background jobs steal I/O	`ionice -c 3 -p $(pgrep backup)`	Set ionice classes: database=RT(1), app=BE(2), backup=Idle(3)
Context switch rate very high	Too many threads competing on few CPUs	`vmstat 1	awk ‘{print $12}’; perf stat -e context-switches`

Hands-on Lab

Objective: Pin CPU affinity and tune I/O scheduler.

# 1. Check CPU topology
lscpu | grep -E "CPU\(s\)|Core|Socket|NUMA"
numactl --hardware

# 2. View current CPU frequencies (if CPUFreq enabled)
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq

# 3. Pin a process to specific CPUs
taskset -c 2,3 stress-ng --cpu 2 --timeout 10 &
# Verify in htop: cores 2 and 3 should spike

# 4. I/O scheduler per device
for dev in /sys/block/sd* /sys/block/nvme*; do
    echo "$dev: $(cat $dev/queue/scheduler 2>/dev/null)"
done

# 5. Change I/O scheduler
echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

# 6. Set I/O priority
sudo ionice -c 2 -n 0 -p $$   # Best-effort, high priority
ionice -p $$

# 7. IRQ affinity
cat /proc/interrupts | grep eth
# echo "ff" > /proc/irq/N/smp_affinity_list  # Spread to all CPUs

Exercises

Pin a CPU-intensive workload to NUMA node 0 CPUs using taskset. Measure performance vs unpinned with perf stat.
Compare database query latency with I/O scheduler none vs mq-deadline on an NVMe SSD using fio and pgbench.
Use ionice -c 3 to run a backup job at idle I/O priority. Verify it doesn’t impact foreground database I/O.
Set CPU performance governor: echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor. Compare latency vs powersave.
Create a udev rule that automatically sets none scheduler for all NVMe devices: ACTION=="add", KERNEL=="nvme*", ATTR{queue/scheduler}="none".

Revision Notes

CPU affinity with taskset and numactl — pin hot threads to specific cores to maximize cache locality.
I/O schedulers: none for NVMe (handles its own queuing), mq-deadline for SATA SSDs, bfq for HDDs needing fairness.
IRQ balancing: spread NIC interrupts across CPUs — use irqbalance for auto, or manual /proc/irq/N/smp_affinity.
CPU frequency governors: performance (max always), powersave (min always), ondemand (dynamic).
I/O priority with ionice: RT (realtime), BE (best-effort, 0-7), Idle. Use Idle for backups, RT for critical DBs.
Context switches: each switch costs ~1-10μs. High context switch rate indicates too many threads for available CPUs.