Cpu_performance

Chapter 51: CPU Performance - Deep Dive

Mastering CPU Performance Monitoring and Tuning

---

51.1 Understanding CPU Architecture

CPU Components

                     CPU Architecture
+------------------------------------------------------------------+
|                                                                   |
|                        CPU Core                                      |
|    +-------------------------------------------------------------+ |
|    |                                                             | |
|    |   +-------------------+                                     | |
|    |   |  Control Unit    |  - Fetch/Decode/Execute          | |
|    |   +-------------------+                                     | |
|    |                                                             | |
|    |   +-------------------+                                     | |
|    |   |  Arithmetic Logic|  - ALU performs calculations      | |
|    |   |  Unit (ALU)      |  - Arithmetic & logical ops       | |
|    |   +-------------------+                                     | |
|    |                                                             | |
|    |   +-------------------+                                     | |
|    |   |  Registers       |  - Fast internal memory            | |
|    |   |  - General       |  - EAX, EBX, ECX, EDX            | |
|    |   |  - Special       |  - PC, SP, FLAGS                  | |
|    |   +-------------------+                                     | |
|    |                                                             | |
|    |   +-------------------+                                     | |
|    |   |  Cache (L1/L2/L3)|  - Fast memory near CPU         | |
|    |   |  - L1: 32KB/core |  - L1: Instruction + Data       | |
|    |   |  - L2: 256KB/core|  - L2: Unified                  | |
|    |   |  - L3: Shared    |  - L3: Shared cache             | |
|    |   +-------------------+                                     | |
|    |                                                             | |
|    +-------------------------------------------------------------+ |
|                              |                                      |
|                              v                                      |
|                        Memory Bus                                   |
|                              |                                      |
|                              v                                      |
|                        RAM (DDR4/DDR5)                             |
|                                                                   |
+------------------------------------------------------------------+

CPU Performance Metrics

                     CPU Performance Metrics
+------------------------------------------------------------------+
|                                                                   |
|    Clock Speed (GHz)                                               |
|    +----------------------------------------------------------+   |
|    |  - Cycles per second                                     |   |
|    |  - Higher = faster execution                            |   |
|    |  - 3.5 GHz = 3.5 billion cycles per second            |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    IPC (Instructions Per Cycle)                                   |
|    +----------------------------------------------------------+   |
|    |  - Instructions completed per clock cycle                |   |
|    |  - Depends on CPU architecture and instruction set       |   |
|    |  - Modern CPUs: 2-4 IPC average                         |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Cores and Threads                                               |
|    +----------------------------------------------------------+   |
|    |  - Physical cores: Actual processing units              |   |
|    |  - Logical cores: Threads (SMT/Hyperthreading)         |   |
|    |  - Example: 8 cores, 16 threads                       |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    Cache Sizes                                                    |
|    +----------------------------------------------------------+   |
|    |  - L1: Fastest, smallest (32KB per core)             |   |
|    |  - L2: Medium (256KB per core)                         |   |
|    |  - L3: Largest, shared (16MB typical)                 |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    TDP (Thermal Design Power)                                     |
|    +----------------------------------------------------------+   |
|    |  - Heat dissipation capability                          |   |
|    |  - Affects cooling requirements                        |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

---

51.2 CPU Monitoring Tools

Comprehensive CPU Monitoring

# =============================================================================
# TOP - Real-time Process Monitor
# =============================================================================

# Basic top
top

# Top with highlighted changes
top -d 1

# Top with color
top -c

# Top in batch mode (for scripting)
top -b -n 1

# Top specific processes
top -p 1234,5678

# Show CPU by core
top
# Press '1' to show all cores

# Sort by CPU
# Press 'P' (shift + p)

# Sort by memory
# Press 'M' (shift + m)

# Show threads
# Press 'H'

# =============================================================================
# HTOP - Enhanced Top
# =============================================================================

# Install
sudo pacman -S htop

# Run htop
htop

# htop shortcuts:
# F2 - Setup (columns, colors)
# F3 - Search
# F4 - Filter
# F5 - Tree view
# F6 - Sort by
# F7 - Nice (-)
# F8 - Nice (+)
# F9 - Kill process

# Specific options
htop -d 10        # 10 second delay
htop -u username  # Show only user processes
htop -p 1234      # Show specific PID

System Monitoring Commands

# =============================================================================
# CPU INFORMATION
# =============================================================================

# Detailed CPU info
lscpu
cat /proc/cpuinfo

# CPU topology
lstopo
lstopo-no-graphics

# Current frequency
cpupower frequency-info

# Monitor frequency
watch -n 1 "grep MHz /proc/cpuinfo"

# =============================================================================
# CPU STATISTICS
# =============================================================================

# Per-CPU statistics (Arch)
mpstat 1
mpstat -P ALL 1

# System activity (Sar)
sar -u 1 5          # CPU usage
sar -q 1 5          # Load average
sar -w 1 5          # Context switches
sar -B 1 5          # Paging statistics

# Virtual memory stats
vmstat 1
vmstat -s

# Detailed stats
vmstat -d

# =============================================================================
# PROCESS MONITORING
# =============================================================================

# List processes with CPU
ps aux --sort=-%cpu | head -20

# Find CPU-intensive processes
top -b -n 1 -o %CPU | head -20

# Thread information
ps -eLf | head -20

# =============================================================================
# LOAD AVERAGE
# =============================================================================

# System load
uptime
w

# Detailed load
cat /proc/loadavg

---

51.3 Understanding CPU Metrics

Load Average Explained

                     Load Average Interpretation
+------------------------------------------------------------------+
|                                                                   |
|    Load Average: 1.52 0.48 0.23                                    |
|                     |      |      |                                |
|                     |      |      +-- 23 min average               |
|                     |      +-- 5 min average (0.48)              |
|                     +-- 1 min average (1.52)                      |
|                                                                   |
|    For 4-core system:                                             |
|    +----------------------------------------------------------+   |
|    |  Load < 4  : Underutilized                              |   |
|    |  Load = 4   : Fully utilized                            |   |
|    |  Load > 4   : Overloaded (processes waiting)           |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    CPU States:                                                    |
|    +----------------------------------------------------------+   |
|    |  user    - Running user space processes                 |   |
|    |  nice    - Running niced user processes                 |   |
|    |  system  - Running kernel processes                     |   |
|    |  iowait  - Waiting for I/O                             |   |
|    |  irq     - Serving interrupts                          |   |
|    |  softirq - Serving soft interrupts                     |   |
|    |  steal   - Stolen by hypervisor (VM)                   |   |
|    |  guest   - Running guest VM                            |   |
|    |  idle    - CPU idle                                    |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

Interpreting vmstat Output

# vmstat 1 output
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1023456  12345  456789    0    0    10     5  100  200  5  2 93  0  0

# procs:
#   r: processes waiting for CPU (run queue)
#   b: processes in uninterruptible sleep (I/O)

# memory:
#   swpd: virtual memory used
#   free: free memory
#   buff: buffers
#   cache: cache

# swap:
#   si: memory swapped in
#   so: memory swapped out

# io:
#   bi: blocks received from block device
#   bo: blocks sent to block device

# system:
#   in: interrupts per second
#   cs: context switches per second

# cpu:
#   us: user time
#   sy: system time
#   id: idle time
#   wa: I/O wait time
#   st: stolen time (VM)

---

51.4 CPU Tuning

CPU Governor

# =============================================================================
# CPU GOVERNOR TYPES
# =============================================================================

# performance  - Always run at maximum frequency
# powersave    - Always run at minimum frequency
# ondemand    - Scale frequency based on demand (default)
# conservative - Scale frequency gradually
# schedutil   - Scheduler-driven (most efficient)

# View available governors
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors

# View current governor
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# Set governor (temporary)
sudo cpupower frequency-set -g performance

# Set for all cores
for cpu in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
    echo performance | sudo tee $cpu
done

# Set specific frequency
sudo cpupower frequency-set -f 3.5GHz

# =============================================================================
# CPU FREQUENCY INFO
# =============================================================================

# Show frequency info
cpupower frequency-info

# Monitor frequency
watch -n 0.1 "cat /proc/cpuinfo | grep MHz"

# =============================================================================
# KERNEL TUNING
# =============================================================================

# Add to /etc/sysctl.conf or /etc/sysctl.d/99-cpu.conf

# Scheduler settings
kernel.sched_migration_cost_ns = 5000000
kernel.sched_latency_ns = 10000000
kernel.sched_min_granularity_ns = 1000000
kernel.sched_wakeup_granularity_ns = 2000000

# Apply
sudo sysctl -p

Process Priority (nice/renice)

# =============================================================================
# NICE VALUES
# =============================================================================

# Range: -20 (highest priority) to +19 (lowest priority)
# Default: 0

# Start process with priority
nice -n 10 command          # Lower priority
nice --15 command           # Highest priority (requires root)

# Examples
nice -n 10 ./backup.sh
nice --15 ./realtime-app

# =============================================================================
# RENICE - Change Priority of Running Process
# =============================================================================

# Change priority
renice +10 -p 1234         # Set to +10
renice -5 -p 1234          # Increase priority

# Change priority of user processes
renice +5 -u username

# Change priority of group processes
renice +5 -g groupname

# View process nice values
ps -eo pid,ni,cmd

---

51.5 CPU Performance Issues

Troubleshooting High CPU Usage

                     High CPU Troubleshooting
+------------------------------------------------------------------+
|                                                                   |
|    1. Identify the culprit                                           |
|    +----------------------------------------------------------+   |
|    |  top -c                    # Show command lines           |   |
|    |  htop                      # Interactive with colors      |   |
|    |  ps aux --sort=-%cpu      # Top CPU consumers            |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    2. Check for runaway processes                                   |
|    +----------------------------------------------------------+   |
|    |  watch -n 1 'ps aux --sort=-%cpu | head -10'          |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    3. Analyze system state                                          |
|    +----------------------------------------------------------+   |
|    |  vmstat 1              # Check CPU states                 |   |
|    |  mpstat 1             # Per-CPU stats                   |   |
|    |  sar -u 1             # Historical CPU usage           |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    4. Check for I/O wait                                            |
|    +----------------------------------------------------------+   |
|    |  vmstat 1              # Check 'wa' column             |   |
|    |  iostat -x 1           # I/O statistics                 |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    5. Check for context switches                                    |
|    +----------------------------------------------------------+   |
|    |  vmstat 1              # Check 'cs' column              |   |
|    |  sar -w 1             # Context switch stats           |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    6. Check for interrupts                                           |
|    +----------------------------------------------------------+   |
|    |  cat /proc/interrupts  # Interrupt count by device        |   |
|    |  mpstat -I ALL 1        # Interrupt per CPU              |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

Common CPU Issues and Solutions

# =============================================================================
# ISSUE: High user CPU
# =============================================================================

# Find CPU-intensive processes
top
# Press P to sort by CPU

# Check application logs
journalctl -xe

# Profile application
perf top
perf record -a -g
perf report

# =============================================================================
# ISSUE: High system CPU (kernel)
# =============================================================================

# Check kernel activity
vmstat 1

# Check for I/O issues
iostat -x 1

# Check interrupts
cat /proc/interrupts

# Check softirqs
cat /proc/softirqs

# =============================================================================
# ISSUE: High iowait
# =============================================================================

# Check disk I/O
iostat -x 1

# Find I/O-heavy processes
iotop

# Check disk usage
df -h

# Check for disk errors
dmesg | grep -i error

# =============================================================================
# ISSUE: High context switches
# =============================================================================

# Check context switch rate
vmstat 1

# Find thread count
ps -eLf | wc -l

# Check process threads
ps -o pid,lwp,nlwp,cmd -p <pid>

# =============================================================================
# ISSUE: CPU steal (virtualized)
# =============================================================================

# Check steal time
vmstat 1

# CPU steal occurs when VM waits for physical CPU
# Solutions:
# - Contact provider about resource allocation
# - Resize to larger instance
# - Check for noisy neighbors

---

51.6 CPU Performance Scripts

Monitoring Script

#!/bin/bash
# cpu-monitor.sh - CPU Performance Monitoring

set -euo pipefail

LOG_FILE="/var/log/cpu-monitor.log"
THRESHOLD=80

log() {
    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

check_cpu() {
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | sed 's/%us,//')

    log "Current CPU usage: ${cpu_usage}%"

    if (( $(echo "$cpu_usage > $THRESHOLD" | bc -l) )); then
        log "WARNING: CPU usage above ${THRESHOLD}%"

        # Get top processes
        log "Top CPU processes:"
        ps aux --sort=-%cpu | head -10 | tee -a "$LOG_FILE"

        # Get load average
        log "Load average: $(uptime | awk -F'load average:' '{print $2}')"

        # Get context switches
        log "Context switches: $(vmstat 1 2 | tail -1 | awk '{print $12}')"
    fi
}

# Main loop
while true; do
    check_cpu
    sleep 60
done

---

51.7 Exam Tips

1. Load average: Understand 1/5/15 min averages
2. CPU states: Know user, system, iowait, idle, steal
3. Tools: Know top, htop, mpstat, vmstat, sar
4. Nice values: -20 is highest priority, +19 lowest
5. CPU governor: performance vs powersave vs ondemand
6. Interrupts: Know how to check /proc/interrupts
7. vmstat: Interpret all columns correctly
8. Troubleshoot: High CPU = find process → analyze → fix
9. Virtualization: Understand CPU steal time
10. Perf tool: Use perf for profiling

---

Summary

In this chapter, you learned:

• ✅ CPU architecture and components
• ✅ CPU performance metrics
• ✅ Monitoring tools (top, htop, mpstat, vmstat)
• ✅ Load average interpretation
• ✅ CPU tuning (governors, nice)
• ✅ Troubleshooting high CPU usage
• ✅ CPU performance scripting
• ✅ Best practices

---

Next Chapter

Chapter 52: Memory Performance

---

Last Updated: February 2026