Device Drivers & Kernel Modules

Chapter 9: Device Drivers & Kernel Modules

Learning Objectives

Understand how device drivers work in the kernel
Know how to load, unload, and manage kernel modules
Understand how udev manages devices in userspace
Know how to troubleshoot hardware and driver issues

What is this? (Beginner On-Ramp)

Device drivers are translator programs. They translate generic Linux commands (like ‘read data’) into the specific electrical signals required by a specific piece of hardware (like a Samsung NVMe drive or an Intel network card).

9.1 Device Driver Architecture

    Device Driver Architecture
    ───────────────────────────

    User Application: write(fd, buf, len) to /dev/sda
          │
          │ VFS
          ▼
    ┌─────────────────────────────────────────────────────┐
    │                    KERNEL                            │
    │                                                      │
    │  Block Device Layer (for disks)                     │
    │  ┌───────────────────────────────────────────────┐  │
    │  │  I/O Scheduler (mq-deadline, none, kyber)     │  │
    │  │  Merges and reorders I/O requests             │  │
    │  └───────────────────────────┬───────────────────┘  │
    │                              ↓                       │
    │  ┌───────────────────────────────────────────────┐  │
    │  │             Device Driver                      │  │
    │  │  (e.g., nvme.ko, ahci.ko, virtio_blk.ko)     │  │
    │  │  Translates generic I/O to hardware commands  │  │
    │  └───────────────────────────┬───────────────────┘  │
    └──────────────────────────────│──────────────────────┘
                                   │ DMA / PCI
                                   ▼
                              Physical Hardware
                              (NVMe SSD, SATA drive)

Device Types

Type	Example	Access	File in /dev
Block device	SSD, HDD	Random access, buffered	`/dev/sda`, `/dev/nvme0n1`
Character device	Terminal, serial	Sequential, unbuffered	`/dev/tty`, `/dev/null`
Network device	Ethernet, WiFi	Via sockets, no /dev	`eth0`, `ens3`

# Identify device type (first character of ls -l)
ls -la /dev/sda /dev/tty /dev/null
# brw-rw----  /dev/sda    ← block (b)
# crw-rw-rw-  /dev/tty    ← character (c)
# crw-rw-rw-  /dev/null   ← character (c)

# Major and minor device numbers
ls -la /dev/sda*
# brw-rw---- 1 root disk 8,   0 Jan 15 10:00 /dev/sda    (major=8, minor=0)
# brw-rw---- 1 root disk 8,   1 Jan 15 10:00 /dev/sda1   (major=8, minor=1)

# Major number: identifies the driver
# Minor number: identifies the specific device instance
cat /proc/devices   # List of major numbers and their drivers

9.2 Kernel Modules

Kernel modules are loadable pieces of kernel code — drivers, filesystems, and other extensions that can be loaded/unloaded without rebooting.

# ── Listing Modules ────────────────────────────────────────
lsmod                              # List loaded modules
lsmod | grep ext4                  # Find a specific module
cat /proc/modules                  # Same info, kernel's view

# ── Module Information ─────────────────────────────────────
modinfo ext4                       # Module metadata
modinfo nvme | grep -E "description|depends|version"

# ── Loading and Unloading ──────────────────────────────────
modprobe ext4                      # Load module (with dependencies)
modprobe -r ext4                   # Unload module (with dependents)

insmod /lib/modules/$(uname -r)/kernel/fs/ext4/ext4.ko   # Load specific file
rmmod ext4                         # Unload (no dependency handling)

# ── Module Parameters ──────────────────────────────────────
# Set at load time:
modprobe nvme_core io_queue_depth=1024

# Or in /etc/modprobe.d/nvme.conf:
# options nvme_core io_queue_depth=1024

# View current module parameters
cat /sys/module/nvme_core/parameters/io_queue_depth

# ── Blacklisting Modules (prevent from loading) ────────────
# /etc/modprobe.d/blacklist.conf
# blacklist nouveau        # Blacklist Nvidia open-source driver
# blacklist pcspkr         # Disable PC speaker

# ── Dependencies ──────────────────────────────────────────
modinfo nvme | grep depends
depmod -a                          # Rebuild module dependencies

Module Security

# Check if Secure Boot is enforcing module signing
cat /proc/sys/kernel/modules_disabled    # 1 = no new modules allowed

# Check module signing status
cat /proc/sys/kernel/module_sig_enforce  # 1 = must be signed
grep CONFIG_MODULE_SIG /boot/config-$(uname -r)

# Kernel lockdown (strongest protection)
cat /sys/kernel/security/lockdown
# none  integrity  confidentiality

# Prevent all module loading (security hardening)
echo 1 > /proc/sys/kernel/modules_disabled   # Runtime
# Or kernel parameter: modules_disabled=1

9.3 udev: Device Management in Userspace

udev monitors kernel device events and creates/removes device files in /dev.

    udev Event Flow
    ────────────────

    Hardware connected (or detected at boot)
          │ kernel sends uevent
          ▼
    ┌─────────────────────────────────────────────┐
    │                  udev daemon                 │
    │  Reads rules from /etc/udev/rules.d/         │
    │  and /lib/udev/rules.d/                     │
    └──────────────────────┬──────────────────────┘
                           │ apply matching rules
    ┌──────────────────────▼──────────────────────┐
    │  Actions:                                    │
    │  • Create /dev/sdb, /dev/sdb1               │
    │  • Set permissions and owner                 │
    │  • Create symlinks (/dev/disk/by-uuid/...)   │
    │  • Run scripts (trigger firmware load, etc.) │
    └─────────────────────────────────────────────┘

# Monitor device events in real time
udevadm monitor

# Get device information
udevadm info -a /dev/sda
udevadm info -a /dev/nvme0n1

# Test what rules would fire for a device
udevadm test /sys/block/sda

# Reload udev rules (after editing rules files)
udevadm control --reload-rules
udevadm trigger

# Example udev rule: name a USB network adapter predictably
# /etc/udev/rules.d/70-network.rules
# SUBSYSTEM=="net", ATTR{address}=="aa:bb:cc:dd:ee:ff", NAME="eth-management"

9.4 Block Device and Storage Management

# ── Device Discovery ──────────────────────────────────────
lsblk                              # Tree view of block devices
lsblk -f                           # Show filesystem info
lsblk -d -o name,type,size,model   # Disk info only

# NVMe specific
nvme list                          # List NVMe drives
nvme smart-log /dev/nvme0          # SMART data for NVMe
nvme id-ctrl /dev/nvme0            # Controller info

# ── Storage Performance ────────────────────────────────────
# Check I/O scheduler
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq    ← [none] is active

# Change I/O scheduler
echo "mq-deadline" > /sys/block/nvme0n1/queue/scheduler

# For SSDs/NVMe: 'none' or 'kyber' is best
# For HDDs: 'mq-deadline' or 'bfq' is best

# Check queue depth
cat /sys/block/nvme0n1/queue/nr_requests

# ── Disk Health ───────────────────────────────────────────
smartctl -H /dev/sda               # Quick health check
smartctl -a /dev/sda               # Full SMART data
# Look for: Reallocated_Sector_Ct, Pending_Sector_Count (non-zero = failing)

9.5 Writing a Simple Kernel Module (Educational)

Understanding the structure of a kernel module helps with troubleshooting:

// hello.c - Minimal kernel module
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("Hello World Kernel Module");

static int __init hello_init(void)
{
    printk(KERN_INFO "Hello from the kernel!\n");
    return 0;
}

static void __exit hello_exit(void)
{
    printk(KERN_INFO "Goodbye from the kernel!\n");
}

module_init(hello_init);
module_exit(hello_exit);

# Makefile
obj-m += hello.o

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

# Build and load
make
sudo insmod hello.ko
dmesg | tail -5   # See "Hello from the kernel!"
sudo rmmod hello
dmesg | tail -5   # See "Goodbye from the kernel!"

9.6 Production Scenarios

Scenario 1: NVMe Drive Slow After Reboot

# Check current I/O scheduler (may have reset to default)
cat /sys/block/nvme0n1/queue/scheduler

# Performance difference:
# bfq scheduler: 50MB/s on NVMe (meant for HDDs)
# none scheduler: 3000MB/s on NVMe

# Persistent fix via udev rule:
# /etc/udev/rules.d/60-io-scheduler.rules
# ACTION=="add|change", KERNEL=="nvme*", ATTR{queue/scheduler}="none"
# ACTION=="add|change", KERNEL=="sd*", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"
# ACTION=="add|change", KERNEL=="sd*", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="mq-deadline"

udevadm control --reload-rules
udevadm trigger

Scenario 2: Driver Loading Failure

# dmesg shows: "NVRM: API mismatch: the client has the version 535.x.x"
# Module fails to load due to version mismatch

# Check what's loaded
lsmod | grep nvidia

# Check dependencies
modinfo nvidia | grep depends

# Force module unload (remove all dependents first)
rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia

# Reload with new version
modprobe nvidia

# If still failing, check dmesg for exact error
dmesg | grep -i "nvidia\|NVRM" | tail -20

9.7 Interview Questions

Q1: What is the difference between a built-in module and a loadable module?

Answer: A built-in module is compiled directly into the kernel image (vmlinuz) and is always available from boot — there’s no loading needed. A loadable module (.ko file) is compiled separately and can be inserted and removed from the running kernel with modprobe/rmmod without rebooting. Built-in modules make sense for critical components (like the boot filesystem driver), while loadable modules allow for smaller kernel images and the ability to update drivers without rebooting. The tradeoff: built-in modules are always in memory; loadable modules only use memory when loaded.

Q2: What does udev do and why is it needed?

Answer: udev is the device manager for Linux. When the kernel detects hardware (at boot or hotplug), it generates a uevent. udev receives these events and creates device files in /dev, sets permissions, creates symlinks (like /dev/disk/by-uuid/, /dev/disk/by-path/), loads firmware, and runs custom scripts. Without udev, device files would need to be created manually or via a static /dev populated at boot. udev’s rule system allows flexible device naming (e.g., giving a network interface a consistent name regardless of driver load order).

9.8 Summary

Concept	Key Point
Device drivers	Kernel code that talks to hardware
Kernel modules	Loadable/unloadable drivers (.ko files)
udev	Creates /dev files and manages hotplug
I/O scheduler	Optimizes disk request ordering
modprobe	Load with dependency resolution
modinfo	Show module metadata and parameters

Next Chapter: Chapter 10: sysctl Tuning

Prerequisites

Chapter 1 (Linux Architecture), Chapter 7 (systemd) — understand the kernel/user boundary and module loading.

Advantages & Disadvantages

Advantages: Plug-and-play support for massive hardware variety, loadable modules mean you don’t need to reboot. Disadvantages: Proprietary drivers (like NVIDIA) can cause kernel taint and instability.

Common Mistakes

Compiling a driver against the wrong kernel headers — driver compiled for 5.15 won’t load on 5.19. Always use headers matching uname -r.
Using rmmod while the module is in use — check with lsmod (use count column must be 0 before removal).
Not handling request_irq failures — if IRQ registration fails, the driver must not proceed; unhandled error = kernel oops.
Forgetting to unregister resources in the error path — if probe() fails halfway through, cleanup registered resources or you’ll leak.
Using deprecated kernel APIs — APIs change between versions; check scripts/checkpatch.pl and kernel changelogs.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Module fails to load (unknown symbol)	Module depends on symbol not exported by kernel	`dmesg	tail -20 after insmod; modinfo shows dependencies`
Device not recognized (no /dev entry)	udev rule missing or driver not loaded	`udevadm monitor; lsusb/lspci; dmesg	grep -i error`
Driver causes kernel oops	NULL pointer dereference or memory corruption in driver	`dmesg	grep ‘Oops’; decode with addr2line`
Module loads but device not functional	Wrong IRQ, DMA channel, or I/O port	`dmesg	grep driver_name; cat /proc/interrupts

Hands-on Lab

Objective: Load, inspect, and understand kernel modules.

# 1. List currently loaded modules
lsmod | head -20
# columns: name, size, used-by count, dependents

# 2. Get detailed info about a module
modinfo ext4
modinfo dm_mod

# 3. Load and unload a module
sudo modprobe dummy  # loads dummy network driver
lsmod | grep dummy
ip link show dummy0
sudo modprobe -r dummy

# 4. Check module parameters
ls /sys/module/dm_mod/parameters/
cat /sys/module/dm_mod/parameters/dm_multipath_use_dm_mpath_err

# 5. View interrupt assignments
cat /proc/interrupts | head -20

# 6. Kernel module blacklisting (persistent)
echo "blacklist nouveau" | sudo tee /etc/modprobe.d/disable-nouveau.conf
sudo dracut --force   # rebuild initramfs

Exercises

Write and compile a ‘hello world’ kernel module that prints a message at load and unload. Load it and verify with dmesg.
Use udevadm monitor --environment while plugging in a USB drive. Identify the udev events and which rules match.
Find all PCI devices and their drivers: lspci -k. Identify a device with no driver and research why.
Create a persistent module load configuration: add a module to /etc/modules-load.d/ and verify it loads after reboot.
Use modprobe.d to set a parameter for a module at load time. Verify the parameter took effect via /sys/module/.

Revision Notes

Device drivers are kernel modules (.ko files) that manage hardware — character, block, and network device types.
Module lifecycle: insmod/modprobe → init_module() → use → rmmod → cleanup_module().
modprobe resolves dependencies automatically; insmod does not — prefer modprobe in production.
udev dynamically creates /dev entries when hardware is detected — rules in /etc/udev/rules.d/.
/proc/interrupts shows IRQ assignments; /sys/bus/pci/devices/ shows PCI device tree.
Blacklisting modules: /etc/modprobe.d/blacklist.conf — prevents automatic loading but doesn’t unload if already loaded.