Live_patching
Chapter 85: Linux Kernel Live Patching
Section titled “Chapter 85: Linux Kernel Live Patching”Overview
Section titled “Overview”Live patching allows you to apply security updates and bug fixes to the Linux kernel without rebooting the system. This is critical for production environments where uptime is paramount, such as high-availability servers, cloud infrastructure, and critical business applications. This chapter covers the major live patching solutions (kpatch, kgraft, Canonical Livepatch), how they work, implementation, and best practices.
85.1 Live Patching Overview
Section titled “85.1 Live Patching Overview”How Live Patching Works
Section titled “How Live Patching Works”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING MECHANISM │├─────────────────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Function Replacement Flow │ ││ ├─────────────────────────────────────────────────────────────────┤ ││ │ │ ││ │ Original Function: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ int vulnerable_function(int arg) { │ │ ││ │ │ // Security issue here │ │ ││ │ │ return arg * 2; │ │ ││ │ │ } │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ │ ││ │ ▼ │ ││ │ Live Patch Installation: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ 1. New function compiled with fix │ │ ││ │ │ 2. Patch module loaded into kernel │ │ ││ │ │ 3. ftrace redirects calls to new function │ │ ││ │ │ 4. Original function remains in memory │ │ ││ │ │ (for functions still using it) │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ ││ │ After Patching: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ int vulnerable_function(int arg) { // NEW VERSION │ │ ││ │ │ // Fixed - no security issue │ │ ││ │ │ if (arg < 0) return 0; // Input validation │ │ ││ │ │ return arg * 2; │ │ ││ │ │ } │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Key Technologies: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ │ ││ │ ftrace (Function Tracer): │ ││ │ - Kernel infrastructure for function hooking │ ││ │ - Used to redirect function calls at runtime │ ││ │ - Available in all modern kernels │ ││ │ │ ││ │ Jump Label: │ ││ │ - Static branch optimization │ ││ │ - Allows conditional patching │ ││ │ │ ││ │ Text Patch: │ ││ │ - atomic replacement of kernel code │ ││ │ - Requires careful handling of running code │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘Live Patching Solutions Comparison
Section titled “Live Patching Solutions Comparison”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING SOLUTIONS COMPARISON │├─────────────────────────────────────────────────────────────────────────┤│ ││ ┌────────────────┬─────────────┬─────────────┬─────────────────┐ ││ │ Feature │ kpatch │ kgraft │ Canonical │ ││ │ │ │ │ Livepatch │ ││ ├────────────────┼─────────────┼─────────────┼─────────────────┤ ││ │ │ │ │ │ ││ │ Vendor │ Red Hat │ SUSE │ Canonical/Ub. │ ││ │ │ │ │ │ ││ │ License │ Open Source │ Open Source │ Proprietary │ ││ │ │ │ │ (free tier) │ ││ │ │ │ │ │ ││ │ Integration │ yum/dnf │ zypper │ ua/canonical- │ ││ │ │ │ │ livepatch │ ││ │ │ │ │ │ ││ │ Patching │ Function- │ Function- │ Function- │ ││ │ Method │ level │ level │ level │ ││ │ │ │ │ │ ││ │ Architecture │ ftrace │ kGraft │ ftrace │ ││ │ │ │ │ │ ││ │ Atomic │ Yes │ Yes │ Yes │ ││ │ Updates │ │ │ │ ││ │ │ │ │ │ ││ │ Livepatch │ No │ Yes │ No │ ││ │ Service │ │ │ (cloud-based) │ ││ │ │ │ │ │ ││ │ Cost │ Free │ Paid (SLE) │ Free tier │ ││ │ │ │ │ available │ ││ │ │ │ │ │ ││ └────────────────┴─────────────┴─────────────┴─────────────────┘ ││ ││ What Can Be Patched: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ ✓ Security vulnerabilities (CVEs) │ ││ │ ✓ Critical bug fixes │ ││ │ ✓ Performance optimizations │ ││ │ ✓ │ ││ │ ✗ Cannot add new syscalls │ ││ │ ✗ Cannot change data structures (usually) │ ││ │ ✗ Cannot patch all kernel parts (interrupt handlers, etc.) │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘85.2 kpatch (Red Hat/Fedora)
Section titled “85.2 kpatch (Red Hat/Fedora)”Installation and Setup
Section titled “Installation and Setup”# ============================================================# KPATCH INSTALLATION# ============================================================
# RHEL/CentOSsudo yum install kpatch
# Fedorasudo dnf install kpatch
# Debian/Ubuntu (from source)sudo apt install kpatch-build
# Load kpatch kernel modulesudo modprobe kpatch
# Check if loadedlsmod | grep kpatch
# Enable kpatch service (RHEL 7+)sudo systemctl enable kpatchsudo systemctl start kpatchBuilding a Patch
Section titled “Building a Patch”# ============================================================# BUILDING A KPATCH# ============================================================
# Prerequisites# - kernel-devel package matching running kernel# - gcc, make, flex, bisonsudo yum install kernel-devel-$(uname -r)sudo yum groupinstall "Development Tools"
# Create patch source file# example: fix a simple bug in mm/page_alloc.c
cat > page_alloc.patch << 'EOF'--- a/mm/page_alloc.c+++ b/mm/page_alloc.c@@ -1234,7 +1234,7 @@ static void __free_pages_core(struct page *page, unsigned int order) atomic_long_add(pages, &vm_zone_stat[NR_FREE_PAGES]);
if (tracepoint_enabled)- trace_mm_page_alloc(page, order, migratetype);+ trace_mm_page_alloc(page, order, migratetype, gfp_flags); else page[0].flags = 0;
EOF
# Build the patch modulekpatch-build page_alloc.patch
# Output: page_alloc.ko
# Alternative: Use source tree# kpatch-build -s /path/to/kernel/source patch.cManaging Patches
Section titled “Managing Patches”# ============================================================# KPATCH MANAGEMENT# ============================================================
# Load a patchsudo kpatch load patch.ko
# List loaded patcheskpatch list# Loaded patch modules:# - patch_name (enabled)
# Check patch detailskpatch list -v# Shows: module, version, description
# Auto-load patch at boot# Copy to /etc/kpatch/sudo cp patch.ko /etc/kpatch/
# Unload a patchsudo kpatch unload patch.ko
# Disable auto-loadsudo rm /etc/kpatch/patch.ko
# Check patch statuscat /sys/kernel/kpatch/status85.3 Canonical Livepatch (Ubuntu)
Section titled “85.3 Canonical Livepatch (Ubuntu)”Setup and Configuration
Section titled “Setup and Configuration”# ============================================================# CANONICAL LIVEPATCH SETUP# ============================================================
# Install livepatch clientsudo snap install canonical-livepatch
# Or via apt (older versions)sudo apt install canonical-livepatch
# Enable with token (get from https://auth.livepatch.canonical.com/)sudo canonical-livepatch enable <your-token>
# Check statussudo canonical-livepatch statussudo canonical-livepatch status --verbose
# Check for available patchessudo canonical-livepatch list
# Configuration# /etc/default/canonical-livepatch# or# /var/snap/canonical-livepatch/current/config
# Disable livepatchsudo canonical-livepatch disable
# Update tokensudo canonical-livepatch enable <new-token>Livepatch Commands
Section titled “Livepatch Commands”# ============================================================# LIVEPATCH MANAGEMENT# ============================================================
# Status (detailed)sudo canonical-livepatch status --verbose
# Example output:# Last check: 2024-01-15 10:30:00# Kernel: 5.4.0-200-generic# CPU model: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz# Machine-id: ...# Architecture: x86_64# Paired: yes# Patch state: Applied# Patch version: 5.4.0-200.211# - CVE-2024-1234: Applied (2024-01-10)# - CVE-2024-5678: Applied (2024-01-12)
# Enable/disablesudo canonical-livepatch enable <token>sudo canonical-livepatch disable
# Manual checksudo canonical-livepatch check
# Refreshsudo canonical-livepatch refresh
# Logsjournalctl -u snap.canonical-livepatch.service85.4 kgraft (SUSE)
Section titled “85.4 kgraft (SUSE)”SUSE Live Patching
Section titled “SUSE Live Patching”# ============================================================# KGRALT (SUSE) SETUP# ============================================================
# Install required packagessudo zypper install kgraft-patch
# Check for live patchessudo kgraft patch list
# Apply patchessudo kgraft patch apply <patch-name>
# List applied patchessudo kgraft patch list
# Remove patchessudo kgraft patch remove <patch-name>
# Check statussystemctl status kgraft*
# Configuration# /etc/sysconfig/kgraft85.5 Best Practices
Section titled “85.5 Best Practices”Production Deployment
Section titled “Production Deployment”# ============================================================# LIVE PATCHING BEST PRACTICES# ============================================================
# 1. Testing# Always test in staging before production# Verify patch doesn't break functionality# Test application performance
# 2. Monitoring# Watch system logs after patchingjournalctl -k -fdmesg -w
# Monitor for crashesjournalctl -b -u kernel
# 3. Rollback Plan# Always have a rollback plan# Know how to disable/revert patcheskpatch list --rollbackable# orsudo canonical-livepatch disable
# 4. Schedule# Apply during maintenance windows (even if no reboot needed)# Monitor for 24-48 hours after patching
# 5. Documentation# Document which patches applied# Track CVE numbers# Note any observed issues
# 6. Security# Apply patches promptly (especially critical CVEs)# Don't rely solely on livepatch - plan kernel upgrades
# 7. Limits# Understand what's patchable# Plan for full kernel updates periodically
# Health check after patching# 1. Check dmesg for errors# 2. Verify services running# 3. Test critical applications# 4. Check system performance metrics# 5. Verify livepatch status85.6 Interview Questions
Section titled “85.6 Interview Questions”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING INTERVIEW QUESTIONS │├─────────────────────────────────────────────────────────────────────────┤ │Q1: What is live patching and why is it important? │ │A1: │- Updates kernel without rebooting │- Critical for high-availability systems │- Reduces security vulnerability window │- Maintains uptime while fixing critical issues │- Used for CVE patches in production │ │─────────────────────────────────────────────────────────────────────────┤ │Q2: How does kpatch work? │ │A2: │1. Creates new version of patched function │2. Loads as kernel module │3. Uses ftrace to redirect function calls │4. Original function preserved for compatibility │5. Atomic replacement of function pointers │ │─────────────────────────────────────────────────────────────────────────┤ │Q3: What are the limitations of live patching? │ │A3: │- Can only patch certain kernel components │- Cannot add new syscalls │- Cannot change data structures │- Cannot patch interrupt handlers, some low-level code │- Memory overhead for patch modules │- Requires compatible kernel version │ │─────────────────────────────────────────────────────────────────────────┤ │Q4: Compare kpatch, kgraft, and Canonical Livepatch. │ │A4: │- kpatch: Red Hat, open source, ftrace-based │- kgraft: SUSE, integrated with zypper │- Canonical Livepatch: Ubuntu, cloud-based service, free tier │ │─────────────────────────────────────────────────────────────────────────┤ │Q5: How do you troubleshoot a system after applying a live patch? │ │A5: │1. Check dmesg for errors │2. Check livepatch status │3. Verify system stability │4. Monitor application health │5. Check if patch applied correctly │6. If issues: rollback or reboot to original kernel │ │─────────────────────────────────────────────────────────────────────────┤ │Q6: When would you NOT use live patching? │ │A6: │- When patch requires data structure changes │- When patching interrupt handlers │- During major kernel version upgrades │- If patch introduces new functionality │- When rolling back is more complex than reboot │- In test environments (just reboot instead) │ │─────────────────────────────────────────────────────────────────────────┤ │Q7: What happens if a live patch causes issues? │ │A7: │- Can disable/unload the patch │- May need to reboot to previous kernel │- Should have monitoring in place │- Document issues for vendor │- Can revert to non-patched kernel │ │─────────────────────────────────────────────────────────────────────────┤ │Q8: How does live patching integrate with CI/CD? │ │A8: │- Automatic patch application via tools │- Integration with patch management systems │- Automated testing after patch application │- Monitoring and alerting │- Rollback automation │ │─────────────────────────────────────────────────────────────────────────┤ │Q9: What is the difference between live patching and kernel upgrade? │ │A9: │- Live patching: Function-level fixes, no reboot │- Kernel upgrade: Full kernel version change, requires reboot │- Both important: Live patch for urgent fixes, upgrade for new feat │- Plan regular kernel upgrades even with live patching │ │─────────────────────────────────────────────────────────────────────────┤ │Q10: How do you enable Canonical Livepatch on Ubuntu? │ │A10: │1. Install: sudo snap install canonical-livepatch │2. Get token from https://auth.livepatch.canonical.com/ │3. Enable: sudo canonical-livepatch enable <token> │4. Check status: sudo canonical-livepatch status │ │└─────────────────────────────────────────────────────────────────────────┘Quick Reference
Section titled “Quick Reference”# kpatch (Red Hat/Fedora)kpatch-build patch.csudo kpatch load patch.kokpatch listsudo kpatch unload patch.ko
# Canonical Livepatch (Ubuntu)sudo snap install canonical-livepatchsudo canonical-livepatch enable <token>sudo canonical-livepatch statussudo canonical-livepatch disable
# kgraft (SUSE)sudo zypper install kgraft-patchsudo kgraft patch apply <patch>sudo kgraft patch listSummary
Section titled “Summary”- Live Patching: Fix kernel without reboot
- Solutions: kpatch (Red Hat), kgraft (SUSE), Canonical Livepatch (Ubuntu)
- Technology: ftrace-based function redirection
- Limitations: Can’t change data structures, limited to certain patches
- Best Practice: Test in staging, have rollback plan
Next Chapter
Section titled “Next Chapter”Chapter 86: Container Orchestration Overview
Last Updated: February 2026