Live Patching
Chapter 85: Linux Kernel Live Patching
Section titled “Chapter 85: Linux Kernel Live Patching”Overview
Section titled “Overview”Live patching allows you to apply security updates and bug fixes to the Linux kernel without rebooting the system. This is critical for production environments where uptime is paramount, such as high-availability servers, cloud infrastructure, and critical business applications. This chapter covers the major live patching solutions (kpatch, kgraft, Canonical Livepatch), how they work, implementation, and best practices.
Why This Matters in DevOps/SRE
Section titled “Why This Matters in DevOps/SRE”Live patching is critical for maintaining security in production environments:
- Zero Downtime: Apply security patches without service interruption
- Compliance: Meet security SLAs without maintenance windows
- Cloud Native: Essential for auto-scaling instances that can’t be rebooted
- Critical Systems: Medical, financial, and infrastructure that require 100% uptime
- Patching Cadence: Enables rapid response to CVEs without disruption
Major cloud providers and enterprises use live patching to maintain security compliance while ensuring availability.
85.1 Live Patching Overview
Section titled “85.1 Live Patching Overview”How Live Patching Works
Section titled “How Live Patching Works”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING MECHANISM │├─────────────────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ Function Replacement Flow │ ││ ├─────────────────────────────────────────────────────────────────┤ ││ │ │ ││ │ Original Function: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ int vulnerable_function(int arg) { │ │ ││ │ │ // Security issue here │ │ ││ │ │ return arg * 2; │ │ ││ │ │ } │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ │ ││ │ ▼ │ ││ │ Live Patch Installation: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ 1. New function compiled with fix │ │ ││ │ │ 2. Patch module loaded into kernel │ │ ││ │ │ 3. ftrace redirects calls to new function │ │ ││ │ │ 4. Original function remains in memory │ │ ││ │ │ (for functions still using it) │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ ││ │ After Patching: │ ││ │ ┌─────────────────────────────────────────────────────────┐ │ ││ │ │ int vulnerable_function(int arg) { // NEW VERSION │ │ ││ │ │ // Fixed - no security issue │ │ ││ │ │ if (arg < 0) return 0; // Input validation │ │ ││ │ │ return arg * 2; │ │ ││ │ │ } │ │ ││ │ └─────────────────────────────────────────────────────────┘ │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ ││ Key Technologies: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ │ ││ │ ftrace (Function Tracer): │ ││ │ - Kernel infrastructure for function hooking │ ││ │ - Used to redirect function calls at runtime │ ││ │ - Available in all modern kernels │ ││ │ │ ││ │ Jump Label: │ ││ │ - Static branch optimization │ ││ │ - Allows conditional patching │ ││ │ │ ││ │ Text Patch: │ ││ │ - atomic replacement of kernel code │ ││ │ - Requires careful handling of running code │ ││ │ │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘Live Patching Solutions Comparison
Section titled “Live Patching Solutions Comparison”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING SOLUTIONS COMPARISON │├─────────────────────────────────────────────────────────────────────────┤│ ││ ┌────────────────┬─────────────┬─────────────┬─────────────────┐ ││ │ Feature │ kpatch │ kgraft │ Canonical │ ││ │ │ │ │ Livepatch │ ││ ├────────────────┼─────────────┼─────────────┼─────────────────┤ ││ │ │ │ │ │ ││ │ Vendor │ Red Hat │ SUSE │ Canonical/Ub. │ ││ │ │ │ │ │ ││ │ License │ Open Source │ Open Source │ Proprietary │ ││ │ │ │ │ (free tier) │ ││ │ │ │ │ │ ││ │ Integration │ yum/dnf │ zypper │ ua/canonical- │ ││ │ │ │ │ livepatch │ ││ │ │ │ │ │ ││ │ Patching │ Function- │ Function- │ Function- │ ││ │ Method │ level │ level │ level │ ││ │ │ │ │ │ ││ │ Architecture │ ftrace │ kGraft │ ftrace │ ││ │ │ │ │ │ ││ │ Atomic │ Yes │ Yes │ Yes │ ││ │ Updates │ │ │ │ ││ │ │ │ │ │ ││ │ Livepatch │ No │ Yes │ No │ ││ │ Service │ │ │ (cloud-based) │ ││ │ │ │ │ │ ││ │ Cost │ Free │ Paid (SLE) │ Free tier │ ││ │ │ │ │ available │ ││ │ │ │ │ │ ││ └────────────────┴─────────────┴─────────────┴─────────────────┘ ││ ││ What Can Be Patched: ││ ┌─────────────────────────────────────────────────────────────────┐ ││ │ ✓ Security vulnerabilities (CVEs) │ ││ │ ✓ Critical bug fixes │ ││ │ ✓ Performance optimizations │ ││ │ ✓ │ ││ │ ✗ Cannot add new syscalls │ ││ │ ✗ Cannot change data structures (usually) │ ││ │ ✗ Cannot patch all kernel parts (interrupt handlers, etc.) │ ││ └─────────────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────────────┘85.2 kpatch (Red Hat/Fedora)
Section titled “85.2 kpatch (Red Hat/Fedora)”Installation and Setup
Section titled “Installation and Setup”# ============================================================# KPATCH INSTALLATION# ============================================================
# RHEL/CentOSsudo yum install kpatch
# Fedorasudo dnf install kpatch
# Debian/Ubuntu (from source)sudo apt install kpatch-build
# Load kpatch kernel modulesudo modprobe kpatch
# Check if loadedlsmod | grep kpatch
# Enable kpatch service (RHEL 7+)sudo systemctl enable kpatchsudo systemctl start kpatchBuilding a Patch
Section titled “Building a Patch”# ============================================================# BUILDING A KPATCH# ============================================================
# Prerequisites# - kernel-devel package matching running kernel# - gcc, make, flex, bisonsudo yum install kernel-devel-$(uname -r)sudo yum groupinstall "Development Tools"
# Create patch source file# example: fix a simple bug in mm/page_alloc.c
cat > page_alloc.patch << 'EOF'--- a/mm/page_alloc.c+++ b/mm/page_alloc.c@@ -1234,7 +1234,7 @@ static void __free_pages_core(struct page *page, unsigned int order) atomic_long_add(pages, &vm_zone_stat[NR_FREE_PAGES]);
if (tracepoint_enabled)- trace_mm_page_alloc(page, order, migratetype);+ trace_mm_page_alloc(page, order, migratetype, gfp_flags); else page[0].flags = 0;
EOF
# Build the patch modulekpatch-build page_alloc.patch
# Output: page_alloc.ko
# Alternative: Use source tree# kpatch-build -s /path/to/kernel/source patch.cManaging Patches
Section titled “Managing Patches”# ============================================================# KPATCH MANAGEMENT# ============================================================
# Load a patchsudo kpatch load patch.ko
# List loaded patcheskpatch list# Loaded patch modules:# - patch_name (enabled)
# Check patch detailskpatch list -v# Shows: module, version, description
# Auto-load patch at boot# Copy to /etc/kpatch/sudo cp patch.ko /etc/kpatch/
# Unload a patchsudo kpatch unload patch.ko
# Disable auto-loadsudo rm /etc/kpatch/patch.ko
# Check patch statuscat /sys/kernel/kpatch/status85.3 Canonical Livepatch (Ubuntu)
Section titled “85.3 Canonical Livepatch (Ubuntu)”Setup and Configuration
Section titled “Setup and Configuration”# ============================================================# CANONICAL LIVEPATCH SETUP# ============================================================
# Install livepatch clientsudo snap install canonical-livepatch
# Or via apt (older versions)sudo apt install canonical-livepatch
# Enable with token (get from https://auth.livepatch.canonical.com/)sudo canonical-livepatch enable <your-token>
# Check statussudo canonical-livepatch statussudo canonical-livepatch status --verbose
# Check for available patchessudo canonical-livepatch list
# Configuration# /etc/default/canonical-livepatch# or# /var/snap/canonical-livepatch/current/config
# Disable livepatchsudo canonical-livepatch disable
# Update tokensudo canonical-livepatch enable <new-token>Livepatch Commands
Section titled “Livepatch Commands”# ============================================================# LIVEPATCH MANAGEMENT# ============================================================
# Status (detailed)sudo canonical-livepatch status --verbose
# Example output:# Last check: 2024-01-15 10:30:00# Kernel: 5.4.0-200-generic# CPU model: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz# Machine-id: ...# Architecture: x86_64# Paired: yes# Patch state: Applied# Patch version: 5.4.0-200.211# - CVE-2024-1234: Applied (2024-01-10)# - CVE-2024-5678: Applied (2024-01-12)
# Enable/disablesudo canonical-livepatch enable <token>sudo canonical-livepatch disable
# Manual checksudo canonical-livepatch check
# Refreshsudo canonical-livepatch refresh
# Logsjournalctl -u snap.canonical-livepatch.service85.4 kgraft (SUSE)
Section titled “85.4 kgraft (SUSE)”SUSE Live Patching
Section titled “SUSE Live Patching”# ============================================================# KGRALT (SUSE) SETUP# ============================================================
# Install required packagessudo zypper install kgraft-patch
# Check for live patchessudo kgraft patch list
# Apply patchessudo kgraft patch apply <patch-name>
# List applied patchessudo kgraft patch list
# Remove patchessudo kgraft patch remove <patch-name>
# Check statussystemctl status kgraft*
# Configuration# /etc/sysconfig/kgraft85.5 Best Practices
Section titled “85.5 Best Practices”Production Deployment
Section titled “Production Deployment”# ============================================================# LIVE PATCHING BEST PRACTICES# ============================================================
# 1. Testing# Always test in staging before production# Verify patch doesn't break functionality# Test application performance
# 2. Monitoring# Watch system logs after patchingjournalctl -k -fdmesg -w
# Monitor for crashesjournalctl -b -u kernel
# 3. Rollback Plan# Always have a rollback plan# Know how to disable/revert patcheskpatch list --rollbackable# orsudo canonical-livepatch disable
# 4. Schedule# Apply during maintenance windows (even if no reboot needed)# Monitor for 24-48 hours after patching
# 5. Documentation# Document which patches applied# Track CVE numbers# Note any observed issues
# 6. Security# Apply patches promptly (especially critical CVEs)# Don't rely solely on livepatch - plan kernel upgrades
# 7. Limits# Understand what's patchable# Plan for full kernel updates periodically
# Health check after patching# 1. Check dmesg for errors# 2. Verify services running# 3. Test critical applications# 4. Check system performance metrics# 5. Verify livepatch status85.6 Interview Questions
Section titled “85.6 Interview Questions”┌─────────────────────────────────────────────────────────────────────────┐│ LIVE PATCHING INTERVIEW QUESTIONS │├─────────────────────────────────────────────────────────────────────────┤ │Q1: What is live patching and why is it important? │ │A1: │- Updates kernel without rebooting │- Critical for high-availability systems │- Reduces security vulnerability window │- Maintains uptime while fixing critical issues │- Used for CVE patches in production │ │─────────────────────────────────────────────────────────────────────────┤ │Q2: How does kpatch work? │ │A2: │1. Creates new version of patched function │2. Loads as kernel module │3. Uses ftrace to redirect function calls │4. Original function preserved for compatibility │5. Atomic replacement of function pointers │ │─────────────────────────────────────────────────────────────────────────┤ │Q3: What are the limitations of live patching? │ │A3: │- Can only patch certain kernel components │- Cannot add new syscalls │- Cannot change data structures │- Cannot patch interrupt handlers, some low-level code │- Memory overhead for patch modules │- Requires compatible kernel version │ │─────────────────────────────────────────────────────────────────────────┤ │Q4: Compare kpatch, kgraft, and Canonical Livepatch. │ │A4: │- kpatch: Red Hat, open source, ftrace-based │- kgraft: SUSE, integrated with zypper │- Canonical Livepatch: Ubuntu, cloud-based service, free tier │ │─────────────────────────────────────────────────────────────────────────┤ │Q5: How do you troubleshoot a system after applying a live patch? │ │A5: │1. Check dmesg for errors │2. Check livepatch status │3. Verify system stability │4. Monitor application health │5. Check if patch applied correctly │6. If issues: rollback or reboot to original kernel │ │─────────────────────────────────────────────────────────────────────────┤ │Q6: When would you NOT use live patching? │ │A6: │- When patch requires data structure changes │- When patching interrupt handlers │- During major kernel version upgrades │- If patch introduces new functionality │- When rolling back is more complex than reboot │- In test environments (just reboot instead) │ │─────────────────────────────────────────────────────────────────────────┤ │Q7: What happens if a live patch causes issues? │ │A7: │- Can disable/unload the patch │- May need to reboot to previous kernel │- Should have monitoring in place │- Document issues for vendor │- Can revert to non-patched kernel │ │─────────────────────────────────────────────────────────────────────────┤ │Q8: How does live patching integrate with CI/CD? │ │A8: │- Automatic patch application via tools │- Integration with patch management systems │- Automated testing after patch application │- Monitoring and alerting │- Rollback automation │ │─────────────────────────────────────────────────────────────────────────┤ │Q9: What is the difference between live patching and kernel upgrade? │ │A9: │- Live patching: Function-level fixes, no reboot │- Kernel upgrade: Full kernel version change, requires reboot │- Both important: Live patch for urgent fixes, upgrade for new feat │- Plan regular kernel upgrades even with live patching │ │─────────────────────────────────────────────────────────────────────────┤ │Q10: How do you enable Canonical Livepatch on Ubuntu? │ │A10: │1. Install: sudo snap install canonical-livepatch │2. Get token from https://auth.livepatch.canonical.com/ │3. Enable: sudo canonical-livepatch enable <token> │4. Check status: sudo canonical-livepatch status │ │└─────────────────────────────────────────────────────────────────────────┘Quick Reference
Section titled “Quick Reference”# kpatch (Red Hat/Fedora)kpatch-build patch.csudo kpatch load patch.kokpatch listsudo kpatch unload patch.ko
# Canonical Livepatch (Ubuntu)sudo snap install canonical-livepatchsudo canonical-livepatch enable <token>sudo canonical-livepatch statussudo canonical-livepatch disable
# kgraft (SUSE)sudo zypper install kgraft-patchsudo kgraft patch apply <patch>sudo kgraft patch listCommon Mistakes & Anti-Patterns
Section titled “Common Mistakes & Anti-Patterns”1. Assuming All Patches Can Be Live Patched
Section titled “1. Assuming All Patches Can Be Live Patched”WRONG:
# Assuming any kernel patch can be live patchedkpatch load CVE-2024-1234.patch# Fails: patch modifies data structuresCORRECT:
# Check if patch is live-patch compatible# Only certain types of changes work:# - Function body changes# - Static variable modifications# - Cannot change function signatures or data structures
# Review patch notes for live patch eligibilitycat /var/log/kpatch.logWhy: Not all kernel patches are compatible with live patching.
2. Not Testing Patches Before Production
Section titled “2. Not Testing Patches Before Production”WRONG:
# Applying patches directly to productionsudo kpatch load security-patch.patch# System crash or unexpected behaviorCORRECT:
# Test in staging first1. Create identical staging environment2. Apply patch and monitor3. Check application compatibility4. Verify no performance degradation5. Then apply to production
# Use kpatch list to track patcheskpatch listWhy: Even security patches can introduce regressions.
3. Not Having Rollback Plan
Section titled “3. Not Having Rollback Plan”WRONG:
# Applying patch without backupkpatch load patch.patch# Problem discovered - no way to undoCORRECT:
# Always verify rollback capabilitykpatch list -a # Show all patches
# If issues occurkpatch unload <patch-name>
# Or revert to previous kernelsudo systemctl reboot# Select previous kernel in GRUB
# Keep patch historykpatch historyWhy: Live patches can cause issues; must be able to rollback quickly.
4. Mixing Live Patching Solutions
Section titled “4. Mixing Live Patching Solutions”WRONG:
# Trying to use multiple live patch systemskpatch load patch.patchkgraft load patch.patch # Conflicts!CORRECT:
# Use only one solution per system# kpatch - Red Hat/Fedora# kgraft - SUSE# Canonical Livepatch - Ubuntu
# Check what's installedsystemctl status kpatchsystemctl status kgraftcanonical-livepatch statusWhy: Multiple live patching solutions can conflict.
5. Ignoring Live Patch Limits
Section titled “5. Ignoring Live Patch Limits”WRONG:
# Trying to live patch critical kernel structures# Changes that won't work:# - Adding new syscalls# - Changing function signatures# - Modifying data structure layouts# - Adding new module dependenciesCORRECT:
# Live patches are limited to:# - Bug fixes# - Security patches# - Small code changes
# For major changes, still need reboot# Plan maintenance windows for major updates
# Monitor for applied patcheskpatch listcat /sys/kernel/debug/livepatch/stateWhy: Not all kernel changes can be live patched.
Summary
Section titled “Summary”- Live Patching: Fix kernel without reboot
- Solutions: kpatch (Red Hat), kgraft (SUSE), Canonical Livepatch (Ubuntu)
- Technology: ftrace-based function redirection
- Limitations: Can’t change data structures, limited to certain patches
- Best Practice: Test in staging, have rollback plan
Next Chapter
Section titled “Next Chapter”Chapter 86: Container Orchestration Overview
Last Updated: February 2026