Live Kernel Patching, Kexec & Crash Dumps

Chapter 46: Live Kernel Patching, Kexec & Crash Dumps

Learning Objectives

Apply live kernel patches without rebooting using kpatch/livepatch
Use kexec for fast reboots and kernel switching
Configure kdump for kernel crash dump collection
Analyze kernel panics with crash utility

What is this? (Beginner On-Ramp)

Live kernel patching is a magic trick that applies critical security updates to the core operating system without needing to reboot the server. This is essential for systems that require 100% uptime and cannot afford even a minute of restart downtime.

46.1 Live Kernel Patching

    The Problem: Kernel CVEs Require Reboots
    ──────────────────────────────────────────

    Critical kernel CVE published → Patch available
    Traditional: Install patch → REBOOT (downtime!)
    Production: "We can't reboot our primary DB at 2pm"
    Result: Systems stay vulnerable for weeks waiting for maintenance window

    Live Kernel Patching:
    ──────────────────────
    Apply security patch to RUNNING kernel in memory
    No reboot required
    Service continues without interruption

    How It Works:
    ─────────────
    1. Compile a kernel module (.ko) with the patched function
    2. Use ftrace to redirect old function to new function
    3. Old function: patched code runs for future calls
    4. In-progress calls: finish with old code (atomic swap)

    Tools:
    • kpatch: Red Hat's implementation
    • livepatch: Canonical's implementation (Ubuntu)
    • kGraft: SUSE's implementation
    • KLP (Kernel Live Patching): Upstream kernel framework

46.2 kpatch (Red Hat/CentOS)

# ── Install kpatch ────────────────────────────────────────
yum install kpatch kpatch-runtime

# ── Apply Official kpatch Module ──────────────────────────
# Red Hat provides pre-built patches via RHSM
subscription-manager attach --pool=POOL_ID   # kpatch-patch subscription

# Install patch for current kernel
yum install "kpatch-patch-$(uname -r | sed 's/-/_/g')"

# Apply the patch
kpatch load /usr/lib/kpatch/$(uname -r)/*.ko

# Verify patch is applied
kpatch list
# Loaded patch modules:
# kpatch-patch-5_14_0-70_13_1-1-2.el9_0.x86_64

# Patch is live — no reboot needed!

# ── Building a Custom kpatch ──────────────────────────────
# Install build dependencies
yum install kpatch-build

# Get kernel source matching your running kernel
yumdownloader --source kernel-$(uname -r)
rpm -ivh kernel-*.src.rpm

# Create patch file
diff -u original_file.c patched_file.c > my-fix.patch

# Build
kpatch-build -t vmlinux my-fix.patch
# Generates: kpatch-my-fix.ko

# Apply
kpatch load kpatch-my-fix.ko

# Make persistent (survive reboot)
kpatch install kpatch-my-fix.ko

# ── Ubuntu Livepatch ──────────────────────────────────────
# Canonical's equivalent (cloud subscription)
snap install canonical-livepatch
canonical-livepatch enable <token>

# Check status
canonical-livepatch status --verbose

46.3 kexec: Fast Kernel Reboots

# kexec loads a new kernel into memory without going through BIOS/UEFI
# Reboot time: ~10-30 seconds (vs 2-5 minutes for full hardware reset)
# Useful for: quick kernel updates when reboot IS needed, consistent boot

# ── Basic kexec Usage ─────────────────────────────────────
# Install
yum install kexec-tools

# Load new kernel into memory
kexec -l /boot/vmlinuz-new \
    --initrd=/boot/initramfs-new.img \
    --command-line="root=/dev/sda1 ro quiet"

# Execute (triggers fast reboot)
kexec -e
# Or: systemctl kexec  (systemd-aware)

# ── Automated kexec on kernel update ──────────────────────
# /etc/kexec.conf (some distributions)
kexec_load=1     # Load new kernel on install

# ── kexec in Containers (Kata Containers) ─────────────────
# Kata uses kexec to boot lightweight VM kernels for each container
# Combines container speed with VM isolation

46.4 kdump: Kernel Crash Dump Collection

# kdump captures RAM contents when kernel panics
# Allows post-mortem debugging of crashes

# ── Setup kdump ───────────────────────────────────────────
yum install kexec-tools

# Reserve memory for kdump kernel (in /etc/default/grub or grubby)
# GRUB_CMDLINE_LINUX="... crashkernel=256M"
grubby --update-kernel=ALL --args="crashkernel=256M"
grub2-mkconfig -o /boot/grub2/grub.cfg

# Enable and start
systemctl enable --now kdump

# Verify
kdumpctl status
# kdump:  configured

# ── Configure dump location ───────────────────────────────
# /etc/kdump.conf
path /var/crash                    # Local dump
# Or dump to NFS:
# nfs 10.0.1.100:/exports/kdump
# Or to SSH:
# ssh root@dump-server.internal
# Or to raw disk:
# raw /dev/sdb

# Compression (important — dumps can be huge)
core_collector makedumpfile -l --message-level 1 -d 31
# -d 31: dump pages with no data (saves space)
# -l: lzo compression

# ── Trigger a test dump ───────────────────────────────────
# (Only on a test system!)
echo c > /proc/sysrq-trigger    # Force kernel panic

# Dump appears in /var/crash/YYYY-MM-DD-HH:MM/vmcore

# ── Analyze crash dump with `crash` ───────────────────────
yum install crash kernel-debuginfo-$(uname -r)

crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux \
      /var/crash/2024-01-15-14:30/vmcore

# Inside crash shell:
# bt           → backtrace at time of crash
# log          → kernel ring buffer (dmesg) at crash
# ps           → running processes at crash
# vm           → virtual memory stats
# net          → network state
# foreach bt   → backtrace for every thread
# sym <address> → symbol name for memory address

46.5 Analyzing a Kernel Panic

    Reading a Kernel Panic Message
    ─────────────────────────────────

    Kernel panic - not syncing: fatal exception in interrupt
    CPU: 3 PID: 1234 Comm: nginx Tainted: G 5.15.0-91-generic
    Call Trace:
    [<ffffffff8106b370>] ? panic+0x12e/0x2c5
    [<ffffffff81072c8c>] ? oops_end+0xbc/0x100
    [<ffffffff81073015>] ? die+0x65/0x90
    ...

    Decoding:
    ──────────
    • "not syncing" = system halted, can't write more logs
    • CPU 3, PID 1234 (nginx) = which CPU/process triggered it
    • "Tainted: G" = proprietary module loaded (G=signed)
    • Call Trace = function call stack (read bottom-up)
    • Address [<ffffffff8106b370>] = kernel virtual address

    Decode addresses with: addr2line / gdb / crash
    addr2line -e /usr/lib/debug/.build-id/.../vmlinux ffffffff8106b370

46.6 Interview Questions

Q1: What is live kernel patching and what are its limitations?

Answer: Live kernel patching (kpatch, livepatch) applies security patches to a running kernel without rebooting — the patch is a kernel module that redirects function calls from the vulnerable function to the patched version using ftrace. This allows applying critical security patches (e.g., dirty pipe, dirty COW) while maintaining uptime. Limitations: (1) Not all patches are live-patchable — structural changes (new data structures, changed function signatures) require reboot. Only carefully crafted patches qualify. (2) Vendor-specific — Red Hat’s kpatch patches work on RHEL, Canonical’s livepatch on Ubuntu. (3) Licensing — commercial subscriptions required (RHEL, Ubuntu Pro). (4) Still need planned reboots — live patches are a bridge to the next maintenance window, not a permanent solution. The patch survives reboots but you’ll eventually reboot to a fully updated kernel.

46.7 Summary

Tool	Purpose
kpatch	Red Hat live kernel patching
livepatch	Ubuntu live kernel patching
kexec	Fast kernel reboot (no BIOS)
kdump	Capture kernel crash dumps
crash	Analyze kernel crash dumps
makedumpfile	Compress and filter crash dumps

Next Chapter: Chapter 47: Kernel Compilation & Custom Modules

Prerequisites

Chapter 9 (Device Drivers), Chapter 47 (Kernel Compilation).

Advantages & Disadvantages

Advantages: Fixes critical CVEs instantly without dropping active user connections (zero downtime). Disadvantages: Patches are temporary (must reboot eventually), highly specific to exact kernel versions.

Common Mistakes

Attempting to live-patch a massive architectural kernel change (live patching is only for small security fixes).
Leaving live patches running indefinitely without eventually rebooting into a fully updated kernel.
Applying a live patch built for a different exact kernel version string.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
kpatch module fails to load	Kernel version mismatch or incompatible patch	`dmesg	tail; kpatch list`
System panics during patch application	Functions in use could not be safely redirected	`Review vmcore / crash dump`	Wait for system load to drop before applying complex live patches

Hands-on Lab

Objective: Inspect live kernel patching status.

# 1. Check if kernel supports live patching
cat /boot/config-$(uname -r) | grep CONFIG_LIVEPATCH

# 2. View loaded live patches (if using kpatch)
# kpatch list
# or
# ls -l /sys/kernel/livepatch/

Exercises

Research how ftrace (Function Tracer) is utilized by the Linux kernel to enable live patching.
Compare the pros and cons of Canonical Livepatch vs. Red Hat kpatch.

Revision Notes

Live patching redirects execution from a vulnerable function to a patched function in memory without rebooting.
It relies on ftrace and kernel modules.
kexec allows booting a new kernel directly from the current kernel, bypassing hardware POST/BIOS (useful for fast reboots).
kdump uses kexec to boot into a crash kernel to capture a memory dump on kernel panic.