perf, eBPF & Linux Observability Tools

Chapter 24: perf, eBPF & Linux Observability Tools

Learning Objectives

Use perf for CPU profiling and performance analysis
Understand eBPF and how it enables powerful, safe kernel observability
Use BCC and bpftrace for production tracing
Apply the USE method to diagnose performance problems

What is this? (Beginner On-Ramp)

Profiling is like putting a microscope on a running program. Using tools like perf, you can see exactly which line of code or which kernel function is consuming all the CPU time, allowing you to fix the exact cause of slowness.

24.1 The USE Method

USE (Utilization, Saturation, Errors) is a performance methodology by Brendan Gregg for systematically diagnosing performance problems.

    USE Method — For Every Resource, Check:
    ─────────────────────────────────────────

    ┌────────────────────────────────────────────────────┐
    │  Utilization: How much of the resource is being    │
    │               used? (busy time % or volume used)   │
    │                                                    │
    │  Saturation: Is work queuing up waiting for the    │
    │              resource? (queue depth, wait time)    │
    │                                                    │
    │  Errors:     Are there errors occurring?           │
    │              (error events, failed operations)     │
    └────────────────────────────────────────────────────┘

    Resources to check:
    ┌──────────┬──────────────────┬──────────────────────┐
    │ Resource │ Utilization      │ Saturation           │
    ├──────────┼──────────────────┼──────────────────────┤
    │ CPUs     │ top %usr+%sys    │ load average / nproc │
    │ Memory   │ free -h          │ swap activity        │
    │ Disk I/O │ iostat %util     │ await (latency)      │
    │ Network  │ sar -n DEV       │ /proc/net/softnet    │
    │ CPU sched│ vmstat r column  │ run queue length     │
    └──────────┴──────────────────┴──────────────────────┘

24.2 perf: Performance Counters

perf is the Linux performance analysis tool. It uses hardware performance counters (PMCs) and kernel tracepoints.

# ── perf stat: Performance Summary ───────────────────────
perf stat ./myapp
perf stat -a sleep 5          # System-wide, 5 seconds

# Output:
#  Performance counter stats for './myapp':
#    1,234,567,890  cycles              # CPU cycles
#      987,654,321  instructions        # CPU instructions
#               0.80  insns per cycle   # IPC (lower = stalled)
#       56,789,012  cache-references
#        5,678,901  cache-misses        # 10% miss rate (good if <5%)
#          123,456  branch-misses       # Mispredictions

# Key metric: Instructions Per Cycle (IPC)
# IPC > 2: CPU-bound, efficient
# IPC < 1: Memory-bound, stalled on cache misses

# ── perf record: CPU Profiling ────────────────────────────
# Record CPU profile
perf record -g ./myapp        # With call graph (-g)
perf record -g -a -F 99 sleep 30  # System-wide, 99Hz, 30 seconds

# View the profile
perf report                   # Interactive TUI
perf report --stdio           # Text output

# Generate Flame Graph
perf script | ./FlameGraph/stackcollapse-perf.pl | \
    ./FlameGraph/flamegraph.pl > flamegraph.svg

# ── perf top: Real-Time Top ───────────────────────────────
perf top                      # Real-time hottest functions
perf top -p <PID>             # For specific process

# ── perf trace: System Call Tracer ────────────────────────
perf trace ls /tmp            # Trace system calls of a command
perf trace -p <PID>           # Trace running process

# ── Performance Events ────────────────────────────────────
# List available events
perf list

# Hardware events:
perf stat -e cycles,instructions,cache-misses,branch-misses ./myapp

# Software events:
perf stat -e context-switches,cpu-migrations,page-faults ./myapp

# Kernel tracepoints:
perf stat -e sched:sched_switch,block:block_rq_issue ./myapp

24.3 eBPF: Extended Berkeley Packet Filter

eBPF is a revolutionary technology that allows running sandboxed programs in the kernel without writing kernel modules or changing kernel source.

    eBPF Architecture
    ──────────────────

    ┌─────────────────────────────────────────────────────────┐
    │                      USER SPACE                          │
    │                                                          │
    │  bpftrace / BCC / libbpf programs                       │
    │  Write eBPF programs in C or high-level DSL             │
    └──────────────────────────┬──────────────────────────────┘
                               │ syscall: BPF_PROG_LOAD
                               ▼
    ┌─────────────────────────────────────────────────────────┐
    │                      KERNEL                              │
    │                                                          │
    │  eBPF Verifier: Proves program is safe               │
    │  (no loops, no invalid memory access, bounded)         │
    │         │                                               │
    │         ▼                                               │
    │  JIT Compiler: Compiles eBPF bytecode → native CPU     │
    │         │                                               │
    │         ▼                                               │
    │  Attach to Hook Points:                                 │
    │  • kprobes/kretprobes (kernel functions)                │
    │  • uprobes (user-space functions)                       │
    │  • tracepoints (static kernel instrumentation)          │
    │  • XDP (network packet processing)                      │
    │  • TC (traffic control)                                 │
    │  • LSM hooks (security)                                 │
    │         │                                               │
    │         ▼                                               │
    │  eBPF Maps: Share data between kernel and user space    │
    └─────────────────────────────────────────────────────────┘

    Why eBPF is transformative:
    ✓ Safe: Verified before execution (no kernel panics)
    ✓ Efficient: JIT-compiled to native code
    ✓ No recompile: Dynamic attachment to running kernel
    ✓ Powerful: Access kernel data structures
    ✓ Production-safe: Used in production at Netflix, Facebook, Google

24.4 BCC Tools

BCC (BPF Compiler Collection) provides dozens of ready-made eBPF tools:

# Install BCC tools
apt-get install bpfcc-tools linux-headers-$(uname -r)  # Ubuntu
# Or from source

# Tools are typically at /usr/share/bcc/tools/

# ── System Performance ────────────────────────────────────
execsnoop                  # Trace all exec() calls
opensnoop                  # Trace all open() calls
biolatency                 # Block I/O latency histogram
biosnoop                   # Trace block I/O with latency
ext4slower 10              # Show ext4 ops slower than 10ms
runqlat                    # Run queue latency histogram
cpudist                    # CPU on-CPU time distribution

# ── Networking ────────────────────────────────────────────
tcpconnect                 # Trace TCP connect() calls
tcpaccept                  # Trace TCP accept() calls
tcpretrans                 # Trace TCP retransmits
tcplife                    # TCP connection lifetimes
tcptop                     # Top bandwidth TCP connections

# ── Memory ────────────────────────────────────────────────
memleak -p <PID>           # Detect memory leaks
oomkill                    # Trace OOM killer events
slabratetop                # Kernel slab allocation rate

# ── Profiling ─────────────────────────────────────────────
profile -F 99 30           # CPU profiling at 99Hz, 30 seconds
offcputime -p <PID> 30     # Off-CPU time analysis

# Examples:
# Find slow disk I/O
/usr/share/bcc/tools/biolatency
# Output: latency distribution histogram in microseconds

# Trace all TCP connections from a specific process
/usr/share/bcc/tools/tcpconnect -p $(pidof nginx)
# PID   COMM    IP SADDR        DADDR        DPORT
# 1234  nginx   4  10.0.1.5     10.0.1.10    5432  (postgres!)

24.5 bpftrace

bpftrace is a high-level tracing language for eBPF, like DTrace for Linux.

# Install
apt-get install bpftrace

# ── One-liners ────────────────────────────────────────────
# Trace all new processes
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s(%d): %s\n", comm, pid, str(args->filename)); }'

# Count system calls by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm, pid] = count(); }'

# Trace file opens
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s(%d): %s\n", comm, pid, str(args->filename)); }'

# Block I/O latency histogram
bpftrace -e 'kprobe:blk_account_io_start { @start[arg0] = nsecs; }
kprobe:blk_account_io_done /@start[arg0]/ {
    @["latency(us)"] = hist((nsecs - @start[arg0]) / 1000);
    delete(@start[arg0]);
}'

# Trace slow disk I/O (>10ms)
bpftrace -e 'kprobe:blk_account_io_start { @start[arg0] = nsecs; }
kprobe:blk_account_io_done /@start[arg0]/ {
    $lat = (nsecs - @start[arg0]) / 1000000;
    if ($lat > 10) {
        printf("SLOW I/O: %dms by %s(%d)\n", $lat, comm, pid);
    }
    delete(@start[arg0]);
}'

# Memory allocation tracing
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc {
    @[pid, comm] = sum(arg0);  # Sum malloc sizes by process
}'

# Trace TCP retransmits
bpftrace -e 'kprobe:tcp_retransmit_skb { @[comm, pid] = count(); }'

bpftrace Scripts

#!/usr/bin/env bpftrace

BEGIN {
    printf("Tracing open() calls... Hit Ctrl-C to end.\n");
    printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH");
}

tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{
    @filename[tid] = args->filename;
}

tracepoint:syscalls:sys_exit_open,
tracepoint:syscalls:sys_exit_openat
/@filename[tid]/
{
    $ret = args->ret;
    $fd = $ret >= 0 ? $ret : -1;
    $errno = $ret >= 0 ? 0 : - $ret;

    printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno,
        str(@filename[tid]));
    delete(@filename[tid]);
}

24.6 Flame Graphs

Flame Graphs (invented by Brendan Gregg) are a visualization of profiling data:

# Install FlameGraph tools
git clone https://github.com/brendangregg/FlameGraph

# Method 1: From perf data
perf record -g -F 99 -a -- sleep 30
perf script | ./FlameGraph/stackcollapse-perf.pl > /tmp/stacks.folded
./FlameGraph/flamegraph.pl /tmp/stacks.folded > /tmp/flamegraph.svg

# Method 2: From bpftrace
bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack] = count(); }
interval:s:30 { print(@); exit(); }' | \
    ./FlameGraph/stackcollapse.pl | \
    ./FlameGraph/flamegraph.pl > /tmp/flamegraph.svg

# Reading a Flame Graph:
# X-axis: time (alphabetically sorted function names)
# Y-axis: call stack depth (bottom = on-CPU, top = running)
# Width:  proportion of CPU time in that function
# Wide boxes = hot code paths
# Tall stacks = deep call chains

24.7 Production Observability Toolkit

# Quick performance triage (60-second analysis)
#!/bin/bash
echo "=== 60-Second Performance Analysis ==="

echo "--- uptime / load ---"
uptime

echo "--- CPU (vmstat 1 5) ---"
vmstat 1 5

echo "--- Memory ---"
free -h

echo "--- Disk I/O ---"
iostat -xz 1 3

echo "--- Network ---"
sar -n DEV 1 3

echo "--- Top processes (CPU) ---"
ps aux --sort=-%cpu | head -10

echo "--- Top processes (Memory) ---"
ps aux --sort=-%mem | head -10

echo "--- Kernel messages ---"
dmesg -T | tail -20

echo "=== Analysis complete ==="

24.8 Interview Questions

Q1: What is eBPF and why is it significant for production observability?

Answer: eBPF (extended Berkeley Packet Filter) is a kernel technology that allows running sandboxed programs in the Linux kernel without writing kernel modules or modifying kernel source. It’s significant because: (1) Safe: The eBPF verifier mathematically proves the program cannot crash the kernel before loading. (2) Powerful: Full access to kernel data structures, function arguments, return values, system calls, network packets. (3) Zero overhead when not attached: No performance impact until you actually run a trace. (4) Production-safe: Used at Netflix, Facebook, Google, Cloudflare in production. Tools like Cilium, Falco, Pixie, and many APMs are built on eBPF.

Q2: What is a Flame Graph and how do you interpret it?

Answer: A Flame Graph is a visualization of profiling data. The X-axis represents time (stacks are sorted alphabetically, not temporally — so left-right position doesn’t indicate time sequence). The Y-axis represents stack depth (bottom = on-CPU function, going up = callers). The width of each box represents the proportion of sampled time spent in that function. Reading: (1) Wide boxes at the top = hot leaf functions consuming most CPU. (2) Wide boxes in the middle = functions that span many children. (3) Flat tops = CPU time in that function, not children. (4) Tall spikes = deep call stacks. Focus on the widest boxes at the top of the graph — those are your hotspots to optimize.

24.9 Summary

Tool	Use Case
perf stat	Hardware counter summary
perf record/report	CPU profiling
perf trace	System call tracing
bpftrace	Custom kernel/app tracing
BCC execsnoop	Trace all process exec
BCC biolatency	Disk I/O latency distribution
BCC tcpconnect	Network connection tracing
Flame Graphs	Visualize profiling data

Next Chapter: Chapter 25: Prometheus, Grafana & Alertmanager

Prerequisites

Chapter 23 (Performance Analysis).

Advantages & Disadvantages

Advantages: Pinpoints the exact line of code or kernel function causing slowness. Disadvantages: High overhead if profiling too frequently, FlameGraphs require expertise to read.

Common Mistakes

Running perf record without -g (call graphs), losing the stack traces needed to find the root cause.
Profiling without debug symbols installed, resulting in hexadecimal addresses instead of function names.
Leaving perf recording for too long, generating massive perf.data files that crash perf report.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
perf report shows only hex addresses	Missing debug symbols	`Check package manager for -dbgsym or -debuginfo packages`	Install debug symbols and re-run perf report
perf cannot record (permission denied)	kernel.perf_event_paranoid setting too restrictive	`cat /proc/sys/kernel/perf_event_paranoid`	sudo sysctl -w kernel.perf_event_paranoid=-1

Hands-on Lab

Objective: Generate a flamegraph.

# 1. Record CPU profile
perf record -F 99 -a -g -- sleep 10

# 2. View report in terminal
perf report --stdio | head -40

# 3. Generate flamegraph (requires FlameGraph scripts)
# perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

Exercises

Write a CPU-heavy Python or C program. Use perf stat to measure its cache misses and instruction count.
Generate a CPU flamegraph of the entire system while running a load test, and identify the hottest kernel function.

Revision Notes

perf uses hardware performance counters (PMCs) to measure events precisely.
Sampling (perf record) is low-overhead; Tracing every event can have high overhead.
Flamegraphs visualize stack traces: width = time on CPU, y-axis = call stack depth.
Install debug symbols for accurate function names.