Linux Architecture & Kernel Overview

Chapter 1: Linux Architecture & Kernel Overview

Learning Objectives

By the end of this chapter, you will:

• Understand the complete layered architecture of a Linux system
• Know what the kernel is, what it does, and why it exists
• Understand the distinction between user space and kernel space
• Understand system call interface and why it matters for security
• Be able to explain Linux internals in a senior engineering interview

Prerequisites

• Basic Linux command-line familiarity (see Linux SysAdmin Guide)
• Understanding of processes and files at a user level

---

What is this? (Beginner On-Ramp)

The Linux architecture is how the operating system is structured. Think of it like a restaurant: the hardware is the kitchen, the applications are the customers, and the Kernel is the waiter taking orders (system calls) and bringing back results. The kernel manages everything so applications don’t have to worry about the underlying hardware.

1.1 What is the Linux Kernel?

The Linux kernel is the core software that sits between hardware and user applications. It is not the entire operating system — it is the fundamental component that manages hardware resources and provides services to user-space programs.

                    The Linux System Stack

    ┌─────────────────────────────────────────────────────┐
    │                  USER APPLICATIONS                   │
    │   bash  nginx  postgres  python  java  ssh  vim     │
    └──────────────────────┬──────────────────────────────┘
                           │  System Calls (read, write, fork, etc.)
    ┌──────────────────────▼──────────────────────────────┐
    │               C STANDARD LIBRARY (glibc)             │
    │   printf() → write()  malloc() → mmap()             │
    └──────────────────────┬──────────────────────────────┘
                           │
    ════════════════════════════════════════════════════════
                    USER SPACE / KERNEL SPACE BOUNDARY
    ════════════════════════════════════════════════════════
                           │
    ┌──────────────────────▼──────────────────────────────┐
    │                   LINUX KERNEL                       │
    │                                                      │
    │  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
    │  │ Process  │  │ Memory   │  │   VFS (Virtual   │  │
    │  │ Scheduler│  │ Manager  │  │   File System)   │  │
    │  └──────────┘  └──────────┘  └──────────────────┘  │
    │  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
    │  │ Network  │  │ Device   │  │   IPC (Pipes,    │  │
    │  │  Stack   │  │ Drivers  │  │   Sockets, SHM)  │  │
    │  └──────────┘  └──────────┘  └──────────────────┘  │
    └──────────────────────┬──────────────────────────────┘
                           │
    ┌──────────────────────▼──────────────────────────────┐
    │                    HARDWARE                          │
    │   CPU  •  RAM  •  Disk  •  NIC  •  USB  •  GPU     │
    └─────────────────────────────────────────────────────┘

Why Does the Kernel Exist?

Without a kernel, every application would need to:

1. Write its own disk drivers
2. Manage CPU scheduling itself
3. Handle hardware conflicts with other programs

The kernel solves this by being the single trusted intermediary between hardware and software.

Key insight: The kernel was invented to solve the problem of safe resource sharing between multiple untrusted programs on shared hardware.

---

1.2 Kernel Space vs User Space

This is the most fundamental concept in Linux architecture. Everything in Linux runs in one of two spaces:

    ┌─────────────────────────────────────────────────────────┐
    │                     USER SPACE                           │
    │                  (Ring 3 — Low Privilege)                │
    │                                                          │
    │   • Applications run here (nginx, bash, python)         │
    │   • Cannot directly access hardware                     │
    │   • Cannot directly access kernel memory                │
    │   • Limited CPU instructions available                  │
    │   • Memory is isolated per-process (virtual memory)     │
    │   • A crash here kills only that process                │
    │                                                          │
    └──────────────────────┬───────────────────────────────────┘
                           │ System Call Interface
                           │ (only crossing point)
    ┌──────────────────────▼───────────────────────────────────┐
    │                    KERNEL SPACE                           │
    │                  (Ring 0 — Full Privilege)               │
    │                                                          │
    │   • Kernel code runs here                               │
    │   • Full hardware access                                │
    │   • Full memory access                                  │
    │   • All CPU instructions available                      │
    │   • Shared across all processes                         │
    │   • A crash here = kernel panic = system halt           │
    │                                                          │
    └───────────────────────────────────────────────────────────┘

CPU Protection Rings

Modern x86 CPUs implement hardware protection rings (0–3). Linux uses only two:

Ring	Name	Used By	Privileges
Ring 0	Kernel Mode	Linux Kernel	Full hardware access, all instructions
Ring 3	User Mode	All user processes	Restricted, no direct hardware access

Why does this matter in production?

• When an application crashes (SIGSEGV), the kernel catches it and kills only that process
• When kernel code crashes, the entire system panics
• Security exploits that escape to Ring 0 (container escapes, privilege escalation) are critical vulnerabilities

---

1.3 System Calls: The Only Door to the Kernel

A system call (syscall) is the only legitimate way for user-space code to request kernel services. When nginx wants to read a file, it doesn’t access the disk directly — it calls read(), which triggers a syscall.

    Application: read(fd, buf, count)
           │
           │  1. CPU switches to Ring 0
           │  2. Kernel validates arguments
           │  3. Kernel performs the operation
           │  4. CPU switches back to Ring 3
           │  5. Return value passed back
           ▼
    Result returned to application

Common System Calls

Category	System Calls	Description
File I/O	open, read, write, close, lseek	File operations
Process	fork, exec, exit, wait, kill	Process management
Memory	mmap, munmap, brk, mprotect	Memory management
Network	socket, bind, listen, accept, connect	Network I/O
Time	clock_gettime, nanosleep, timer_create	Timing
IPC	pipe, socket, shmget, semop	Inter-process communication

Tracing System Calls in Production

# Trace all syscalls made by a process
strace -p <PID>

# Count syscalls and show summary (great for performance analysis)
strace -c -p <PID>

# Trace specific syscalls only
strace -e trace=read,write,open,close nginx

# Trace a new command and show timing
strace -T -e trace=file ls /etc

# Filter by return value (find failing syscalls)
strace -e trace=open -e 'fault=open:retval=-1' nginx 2>&1 | grep "= -1"

# Production use: trace syscalls of a hung process
strace -p <PID> -s 1024 -v -o /tmp/strace_output.txt

Production Example: A PostgreSQL process was consuming 100% CPU. Using strace -c -p <PID> revealed it was making millions of futex syscalls — indicating a lock contention bug in a library.

---

1.4 The Monolithic vs Microkernel Debate

Linux uses a monolithic kernel architecture. This is a critical architectural choice with major trade-offs.

    MONOLITHIC KERNEL (Linux)          MICROKERNEL (Minix, QNX)
    ─────────────────────────          ─────────────────────────
    ┌───────────────────────┐          ┌───────────────────────┐
    │      USER SPACE       │          │      USER SPACE       │
    │  App A   App B   App C│          │ App A  App B  App C   │
    └────────┬──────────────┘          │ FS Drv  Net Drv  etc  │
             │ syscall                  └────────┬──────────────┘
    ┌────────▼──────────────┐                   │ IPC
    │     KERNEL SPACE      │          ┌─────────▼─────────────┐
    │  ┌─────────────────┐  │          │    MICROKERNEL         │
    │  │  Scheduler      │  │          │ (minimal: IPC + sched) │
    │  │  Memory Mgr     │  │          └───────────────────────┘
    │  │  File Systems   │  │
    │  │  Network Stack  │  │
    │  │  Device Drivers │  │
    │  └─────────────────┘  │
    └───────────────────────┘

    PROS:                              PROS:
    + Fast (direct function calls)     + Fault isolation (driver crash
    + Easier to share data               doesn't crash kernel)
    + Mature ecosystem                 + Smaller trusted computing base

    CONS:                              CONS:
    - Driver bug = kernel panic        - IPC overhead (slower)
    - Large attack surface            - Complex to implement correctly
    - Hard to isolate failures

Why Linux chose monolithic: Performance. The cost of IPC (context switches, data copying) for every driver call would make Linux too slow for general-purpose computing. Linux mitigates the isolation problem through kernel modules (loadable drivers) and strict code review.

---

1.5 The Linux Kernel’s Major Subsystems

    ┌──────────────────────────────────────────────────────────────┐
    │                    LINUX KERNEL                              │
    │                                                              │
    │  ┌─────────────┐   ┌────────────────┐   ┌───────────────┐  │
    │  │   PROCESS   │   │    MEMORY      │   │   FILE SYS    │  │
    │  │  SCHEDULER  │   │   MANAGER      │   │   (VFS)       │  │
    │  │             │   │                │   │               │  │
    │  │ • CFS       │   │ • Virtual mem  │   │ • ext4, XFS   │  │
    │  │ • Real-time │   │ • Page cache   │   │ • tmpfs       │  │
    │  │ • Cgroups   │   │ • Slab alloc   │   │ • procfs      │  │
    │  │ • Namespaces│   │ • NUMA         │   │ • sysfs       │  │
    │  └──────┬──────┘   └───────┬────────┘   └───────┬───────┘  │
    │         │                  │                     │          │
    │  ┌──────▼──────┐   ┌───────▼────────┐   ┌───────▼───────┐  │
    │  │   NETWORK   │   │    DEVICE      │   │    SECURITY   │  │
    │  │    STACK    │   │   DRIVERS      │   │   SUBSYSTEM   │  │
    │  │             │   │                │   │               │  │
    │  │ • TCP/IP    │   │ • Block devs   │   │ • LSM (SELinux│  │
    │  │ • netfilter │   │ • Char devs    │   │   AppArmor)   │  │
    │  │ • eBPF/XDP  │   │ • Network devs │   │ • Capabilities│  │
    │  │ • Sockets   │   │ • USB/PCIe     │   │ • Seccomp     │  │
    │  └─────────────┘   └────────────────┘   └───────────────┘  │
    │                                                              │
    │  ┌──────────────────────────────────────────────────────┐   │
    │  │                   ARCH-SPECIFIC CODE                  │   │
    │  │          x86_64  •  ARM64  •  RISC-V  •  s390x       │   │
    │  └──────────────────────────────────────────────────────┘   │
    └──────────────────────────────────────────────────────────────┘

Process Scheduler (CFS)

The Completely Fair Scheduler (CFS) decides which process runs on which CPU at any given moment. It uses a red-black tree to track virtual runtimes and gives each process a “fair” share. (Covered in depth in Chapter 3.)

Memory Manager

Manages the mapping of virtual addresses to physical RAM. Implements the page cache (frequently accessed file data cached in RAM), the slab allocator (efficient kernel object allocation), and handles memory pressure. (Covered in Chapter 4.)

VFS (Virtual File System)

An abstraction layer that allows Linux to support many different filesystem types (ext4, XFS, Btrfs, NFS, FUSE) through a single unified API. Applications just call read() — VFS routes the call to the right filesystem driver.

Network Stack

Implements TCP/IP from the bottom up: network device drivers → L2 (Ethernet) → L3 (IP, routing) → L4 (TCP/UDP) → sockets. Also contains netfilter (iptables/nftables), eBPF hooks, and the XDP fast path.

Linux Security Module (LSM)

A framework that allows security modules (SELinux, AppArmor) to hook into kernel operations and enforce mandatory access control. (Covered extensively in Part 3.)

---

1.6 The /proc and /sys Filesystems

These two pseudo-filesystems are your live window into the kernel. They don’t exist on disk — the kernel generates their content in real time.

/proc: Process and System Information

# View kernel version
cat /proc/version

# View CPU information
cat /proc/cpuinfo

# View memory information
cat /proc/meminfo

# View all running processes (each directory is a PID)
ls /proc/ | grep '^[0-9]'

# View a specific process's memory maps
cat /proc/1234/maps

# View a process's open file descriptors
ls -la /proc/1234/fd/

# View network connections (same as ss/netstat)
cat /proc/net/tcp

# View kernel parameters (same as sysctl -a)
cat /proc/sys/vm/swappiness

# View system-wide open file stats
cat /proc/sys/fs/file-nr

# View interrupt statistics per CPU
cat /proc/interrupts

# View system uptime
cat /proc/uptime

/sys: Hardware and Driver Information

# View all block devices
ls /sys/block/

# View NVMe device information
cat /sys/block/nvme0n1/queue/scheduler

# View network interface statistics
cat /sys/class/net/eth0/statistics/rx_bytes

# Control a device's power state
echo "on" > /sys/bus/pci/devices/0000:00:01.0/power/control

# View CPU topology
cat /sys/devices/system/cpu/cpu0/topology/core_id

# Set CPU governor (for performance tuning)
echo "performance" > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Production use: During a performance incident, /proc/meminfo, /proc/interrupts, and /proc/net/dev are the first files you examine before touching any monitoring tool.

---

1.7 Kernel Architecture: The Big Picture

How the Kernel Handles an I/O Request

Let’s trace what happens when nginx calls read() to serve a file:

    nginx process calls read(fd, buf, count)
              │
              │ 1. CPU trap → switch to Ring 0
              ▼
    ┌─────────────────────────┐
    │  System Call Handler     │
    │  sys_read()              │
    └──────────┬──────────────┘
               │ 2. File descriptor lookup
               ▼
    ┌─────────────────────────┐
    │  VFS Layer               │
    │  vfs_read()              │
    └──────────┬──────────────┘
               │ 3. Check page cache
               ▼
    ┌─────────────────────────┐     Cache HIT?
    │  Page Cache              │ ────────────────►  Copy data to user buffer
    │  (RAM cache of files)    │                    Return immediately
    └──────────┬──────────────┘
               │ Cache MISS
               ▼
    ┌─────────────────────────┐
    │  Filesystem Driver       │
    │  (ext4_file_read_iter)   │
    └──────────┬──────────────┘
               │ 4. Submit block I/O request
               ▼
    ┌─────────────────────────┐
    │  Block Layer             │
    │  I/O scheduler (mq-deadline│
    └──────────┬──────────────┘
               │ 5. Driver DMA transfer
               ▼
    ┌─────────────────────────┐
    │  NVMe/SATA Driver        │
    │  Hardware DMA            │
    └──────────┬──────────────┘
               │ 6. Hardware interrupt when done
               ▼
    ┌─────────────────────────┐
    │  Return to VFS           │
    │  Update page cache       │
    │  Copy to user buffer     │
    └──────────┬──────────────┘
               │ 7. Return to Ring 3
               ▼
    nginx receives data, serves HTTP response

This entire flow — from syscall to hardware and back — typically takes 50–200 microseconds for an SSD, or <1 microsecond for a page cache hit.

---

1.8 Kernel Version and Features

# Check kernel version
uname -r
# Output: 6.8.0-45-generic

# Detailed kernel info
uname -a

# Check kernel compilation config
zcat /proc/config.gz | grep CONFIG_CGROUPS

# List loaded kernel modules
lsmod

# Show module information
modinfo ext4

# Check kernel command line parameters used at boot
cat /proc/cmdline

Kernel Version Numbering

    6   .   8   .   0   -   45   -   generic
    │       │       │       │         │
    │       │       │       │         └── Distro-specific suffix
    │       │       │       └──────────── ABI version (distro patches)
    │       │       └──────────────────── Patch level
    │       └──────────────────────────── Minor version (new features)
    └──────────────────────────────────── Major version

LTS (Long Term Support) kernels: Production systems should run LTS kernels (e.g., 5.15, 6.1, 6.6). These receive security patches for 2–6 years.

---

1.9 Real-World Production Scenarios

Scenario 1: Diagnosing “System is Slow” Without Tools

When you SSH into a production server with no monitoring tools, the kernel’s pseudo-filesystems give you everything:

# Step 1: Quick health snapshot
echo "=== Load Average ==="
cat /proc/loadavg

echo "=== Memory ==="
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Cached|Buffers|SwapUsed"

echo "=== CPU Stats ==="
cat /proc/stat | head -5

echo "=== Disk I/O ==="
cat /proc/diskstats | grep -E "sda|nvme0n1"

echo "=== Network ==="
cat /proc/net/dev

echo "=== Top syscalls across all processes ==="
for pid in /proc/[0-9]*/; do
    pid_num=$(basename $pid)
    comm=$(cat $pid/comm 2>/dev/null)
    echo "$pid_num $comm"
done | head -20

Scenario 2: Container Escape Attempt Detection

When a container attempts to escape, it typically tries to access kernel interfaces:

# Monitor for suspicious /proc access from containers
# (In practice, this is done with auditd or Falco)
auditctl -w /proc/sysrq-trigger -p wa -k container_escape
auditctl -w /proc/kmsg -p r -k kernel_log_read

# Check if a process is in a namespace (containerized)
ls -la /proc/<PID>/ns/

# Compare against host namespaces
ls -la /proc/1/ns/

---

1.10 Best Practices

Practice	Why	How
Run LTS kernels in production	Security patches + stability	Pin kernel version in provisioning
Monitor kernel messages	Early warning of hardware issues	journalctl -k -f or forward to ELK
Understand kernel parameters before changing	Wrong sysctl can crash the system	Test on staging, document changes
Keep kernel modules minimal	Each module expands attack surface	Audit with lsmod, blacklist unused
Enable kernel ASLR	Mitigates many exploits	echo 2 > /proc/sys/kernel/randomize_va_space

---

1.11 Common Mistakes

❌ Mistake 1: Treating the kernel as a black box
   Problem: Cannot diagnose kernel-level issues
   Fix: Learn to read /proc, /sys, and kernel logs

❌ Mistake 2: Running bleeding-edge kernels in production
   Problem: New bugs introduced with new features
   Fix: Use LTS kernels, test before upgrading

❌ Mistake 3: Ignoring kernel OOM kills
   Problem: Services die silently, blamed on "crashes"
   Fix: Monitor /proc/kmsg or journalctl -k for "OOM killer"

❌ Mistake 4: Compiling custom kernels for every server
   Problem: Hard to maintain, patch, and audit
   Fix: Use distro kernels unless you have specific needs

❌ Mistake 5: Not knowing the difference between kernel and system
   Problem: Confusing "kernel update" with "system upgrade"
   Fix: Understand that kernel, libc, and init are separate components

---

1.12 Troubleshooting

Kernel Panic Recovery

# After a kernel panic, check the crash dump
journalctl -k --boot=-1          # Previous boot kernel logs
journalctl -k -p 0..3            # Current boot kernel errors only

# Check if a kernel oops occurred (non-fatal kernel bug)
dmesg | grep -i "oops\|BUG\|panic\|call trace"

# Check kernel ring buffer for hardware errors
dmesg -T | grep -i "error\|fail\|warn" | tail -50

# Enable kernel crash dumps (kdump) - critical for production
systemctl enable kdump
# After a panic, analyze with:
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

Diagnosing High System CPU

# High %sys means lots of kernel code execution
# Check which syscalls are most frequent
perf stat -e 'syscalls:sys_enter_*' -p <PID> sleep 5

# Or use bpftrace (eBPF-based)
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'

# Check if it's interrupt-driven (high %irq or %softirq)
cat /proc/interrupts | sort -k2 -n -r | head -20

---

1.13 Interview Questions

Conceptual Questions

Q1: What is the difference between kernel space and user space? Why does this separation exist?

Answer: User space is where user applications run with restricted privileges (CPU Ring 3). Kernel space is where the kernel runs with full hardware access (CPU Ring 0). This separation exists for security and stability — if user applications had direct hardware access, a buggy program could corrupt memory, access other processes’ data, or crash the system. The separation enforces isolation. The only way to cross from user space to kernel space is through system calls, which the kernel validates.

Q2: What happens when a process calls malloc() in C?

Answer: malloc() is a C library function (not a syscall). It first checks if the process already has free heap space. If yes, it returns memory without entering the kernel. If more memory is needed, it calls brk() or mmap() (both are real syscalls) to ask the kernel to expand the process’s address space. The kernel allocates virtual memory pages, but actual physical RAM is only allocated when the process first writes to those pages (on-demand paging / copy-on-write).

Q3: What is a system call and how does it work at the hardware level?

Answer: A system call is the mechanism by which user-space programs request kernel services. At the hardware level: (1) The program loads the syscall number into a CPU register (e.g., rax on x86_64) and arguments into other registers. (2) It executes the syscall instruction (or int 0x80 on older 32-bit systems). (3) The CPU switches to Ring 0 (kernel mode) and jumps to the kernel’s syscall handler (determined by the IA32_LSTAR MSR). (4) The kernel validates arguments, performs the operation, writes the return value to rax. (5) The sysret instruction returns to Ring 3 (user mode).

Q4: Explain the role of /proc. Is it a real filesystem?

Answer: /proc is a virtual filesystem generated by the kernel at runtime. It doesn’t exist on disk — the kernel generates the file content on-the-fly when you read from it. It serves two purposes: (1) Exposing kernel and process information (memory maps, file descriptors, CPU stats, etc.) in a human-readable form. (2) Providing a writable interface to tune kernel parameters via /proc/sys/ (equivalent to sysctl). Reading /proc/meminfo actually calls a kernel function that reads current memory accounting variables.

Q5: What is the difference between a monolithic kernel and a microkernel? Why did Linux choose monolithic?

Answer: A monolithic kernel runs all kernel services (drivers, file systems, networking) in a single address space in Ring 0. A microkernel runs only basic primitives (IPC, scheduling) in Ring 0, with other services running as user-space servers communicating via IPC. Linux chose monolithic primarily for performance — every IPC call between user-space servers involves multiple context switches and memory copies. The downside is that a driver bug can crash the entire kernel. Linux mitigates this through strict kernel code review, loadable modules, and future work on eBPF for safe kernel programmability.

Scenario-Based Questions

Q6: A production server is showing high %sys CPU. What does this mean and how do you diagnose it?

Answer: High %sys means the CPU is spending a lot of time executing kernel code (as opposed to user code in %usr, or waiting for I/O in %iowait). Common causes: (1) High syscall rate (many small I/O operations, many context switches), (2) Network packet processing (high packet rates), (3) Lock contention (spinlocks in the kernel). Diagnosis: perf top to see which kernel functions are consuming CPU, strace -c -p <PID> to count syscalls, /proc/interrupts to check interrupt rates, cat /proc/net/dev to check packet rates.

---

1.14 Hands-on Labs

Lab 1: Explore the Kernel via /proc

# 1. Find your shell's PID
echo $$

# 2. Explore your process in /proc
ls /proc/$$
cat /proc/$$/status          # Process status, memory, capabilities
cat /proc/$$/maps            # Virtual memory map
cat /proc/$$/limits          # Resource limits
ls -la /proc/$$/fd/          # Open file descriptors
cat /proc/$$/net/tcp         # TCP connections (in your network namespace)

# 3. Understand the kernel's view of memory
cat /proc/meminfo
# MemTotal: total physical RAM
# MemAvailable: usable RAM (buffers+cache can be reclaimed)
# Cached: file cache (can be reclaimed)
# Buffers: filesystem metadata cache

# 4. Watch syscalls in real time
strace -c sleep 5
# Shows all syscalls, count, and time spent

Lab 2: System Call Tracing

# Trace all file opens in the system (requires root)
strace -e trace=open,openat ls /etc 2>&1 | head -20

# See the actual syscall for a simple write
strace -e trace=write echo "hello"

# Find which syscalls a web server uses
strace -c -p $(pidof nginx | awk '{print $1}') &
sleep 10
kill %1
# Look for: accept4, recvfrom, sendfile, epoll_wait

Lab 3: Kernel Module Exploration

# List all loaded modules
lsmod

# Show module dependencies
modinfo nf_conntrack

# Load a module (example: loop device)
sudo modprobe loop

# Unload a module
sudo modprobe -r loop

# See which modules are built in (not loadable)
cat /lib/modules/$(uname -r)/modules.builtin | grep ext4

---

1.15 Summary

Concept	Key Point
Linux Kernel	Core software mediating hardware access
User/Kernel space	Separation for security and stability
CPU Rings	Hardware enforcement of privilege levels
System Calls	Only legitimate crossing point between spaces
Monolithic kernel	All subsystems in one address space
/proc	Virtual filesystem exposing kernel internals
/sys	Virtual filesystem for hardware/driver interface

1.16 Revision Notes

• Ring 0 = kernel mode (full privileges), Ring 3 = user mode (restricted)
• A syscall causes a CPU mode switch: user → kernel → user
• /proc/<PID>/ contains everything about a process the kernel knows
• Linux is monolithic (not microkernel) for performance reasons
• Kernel crash = kernel panic (system halt); process crash = SIGSEGV (only that process dies)
• dmesg and journalctl -k show kernel log messages

1.17 Related Chapters

• Chapter 2: Boot Process — How the kernel loads
• Chapter 3: Process Lifecycle — How processes work in the kernel
• Chapter 4: Memory Management — Deep dive into virtual memory
• Chapter 5: Cgroups & Namespaces — Container building blocks

1.18 Further Reading

• Linux Kernel Development by Robert Love
• Understanding the Linux Kernel by Bovet & Cesati
• Kernel documentation: https://www.kernel.org/doc/html/latest/
• man 2 syscall — Linux syscall manual

---

Next Chapter: Chapter 2: Boot Process: BIOS → GRUB → Kernel → systemd

---

Last Updated: July 2026

---

Advantages & Disadvantages

Advantages: High performance, strict isolation between user/kernel, open-source transparency. Disadvantages: Steep learning curve, monolithic architecture means a driver bug can crash the whole system.

---

Common Mistakes

• Confusing kernel space and user space: assuming a user-space crash can corrupt the kernel — it cannot (MMU protection). Kernel bugs CAN crash the whole system.
• Thinking “Linux = the kernel”: Linux is the kernel. The OS is GNU/Linux or a distribution. Commands like ls, bash, apt are NOT part of the kernel.
• Treating syscalls as free: every read(), write(), open() is a kernel context switch. Tight loops doing millions of tiny syscalls (e.g., read(fd, buf, 1)) are a major performance anti-pattern — use buffered I/O.
• Ignoring NUMA topology on multi-socket servers: allocating memory on socket 0 from a process running on socket 1 doubles memory latency. Always check numactl --hardware.
• Missing the virtual filesystem layer: developers assume filesystem calls go directly to the disk. Between write() and the disk sit: VFS → page cache → block layer → I/O scheduler → driver. Data on disk only after fsync().

---

Troubleshooting

Symptom	Likely Cause	Diagnosis	Fix
System unresponsive, only Sysrq works	Kernel panic / OOM	dmesg | tail -50	Reboot; analyze kdump
SIGKILL doesn’t kill a process	Process in D (uninterruptible sleep)	ps aux | grep " D "	Fix hung I/O; reboot if stuck
Random segfaults in user apps	Memory corruption or bad hardware	dmesg | grep -i mce; run memtest86	Replace faulty RAM
High sy time in top	Many syscalls (context switches)	perf stat -e context-switches -p PID	Batch syscalls; use io_uring
ENOMEM even with free RAM	Memory fragmentation or cgroup limit	cat /proc/buddyinfo; check cgroup limits	echo 3 > /proc/sys/vm/drop_caches

---

Hands-on Lab

Objective: Map the Linux architecture by inspecting a running system.

# 1. Identify your kernel version and build info
uname -a
cat /proc/version

# 2. List all loaded kernel modules
lsmod | head -20
# Pick one and inspect it:
modinfo $(lsmod | awk 'NR==2{print $1}')

# 3. Count open file descriptors system-wide
cat /proc/sys/fs/file-nr
# Output: opened / 0 / max

# 4. Inspect a process's memory map
cat /proc/$$/maps | head -20
# Notice: text (r-xp), data (rw-p), heap, stack, vdso

# 5. Count syscalls made in 5 seconds
strace -c sleep 5 2>&1

# 6. See kernel virtual address layout (KASLR offset)
sudo cat /proc/kallsyms | grep " T startup_64"

# 7. Inspect interrupts
cat /proc/interrupts | head -10

Expected outcome: you can explain what each section of /proc/PID/maps represents and why a process needs multiple segments.

---

Exercises

1. Find the PID of your shell. Inspect its /proc/PID/ directory — explain the purpose of fd/, maps, status, and cmdline.
2. Write a shell one-liner that counts the total number of open file descriptors across all processes.
3. Using strace, find out which syscall ls /tmp uses to read directory entries. Is it getdents64 or readdir?
4. Use lsmod to find a module with dependencies. Draw the dependency tree for it.
5. Explain why malloc() in user space eventually results in brk() or mmap() syscalls, but not on every call.

---

Revision Notes

• The Linux kernel manages CPU, memory, devices, and provides system calls as the user-kernel interface.
• User space (ring 3) ↔ kernel space (ring 0): separated by privilege levels enforced by CPU hardware.
• System calls are the ONLY legal way for user programs to request kernel services.
• Virtual memory gives each process its own isolated address space — the MMU translates virtual → physical.
• Everything is a file in Linux: regular files, directories, sockets, devices, pipes — all use the same open/read/write/close interface.
• The VFS (Virtual Filesystem Switch) is the abstraction layer that makes ext4, XFS, NFS, procfs all look the same to applications.
• dmesg is your first debugging tool for kernel-level issues.

---

Further Reading

• Linux Kernel Documentation — official kernel docs
• Linux Kernel Development by Robert Love — best intro to internals
• Understanding the Linux Kernel by Bovet & Cesati — deep reference
• Brendan Gregg’s Linux Performance — performance tools map
• man 2 syscalls — complete syscall reference

---

Related Chapters

• Chapter 2: Boot Process — how the kernel starts
• Chapter 3: Process Lifecycle — after the kernel starts
• Chapter 4: Memory Management — virtual memory deep dive
• Chapter 5: Cgroups & Namespaces — kernel isolation primitives

---

Last Updated: July 2026