Skip to content

Process Lifecycle, Threads & Scheduler

Chapter 3: Process Lifecycle, Threads & Scheduler

Section titled “Chapter 3: Process Lifecycle, Threads & Scheduler”
  • Understand how processes are created, executed, and terminated in Linux
  • Understand the difference between processes and threads at the kernel level
  • Understand how the Linux CFS (Completely Fair Scheduler) works
  • Understand process states and why a process might be stuck in each state
  • Know how to diagnose scheduler and process issues in production

A process is simply a running program. The process lifecycle is the story of that program: how it’s born (fork), how it runs and consumes resources (CPU, RAM), and how it eventually dies (exit) and is cleaned up by the operating system.

A process is an instance of a running program. It is the fundamental unit of isolation in Linux.

What Makes a Process?
─────────────────────
┌─────────────────────────────────────────────────────────────┐
│ PROCESS │
│ │
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
│ │ ADDRESS SPACE │ │ RESOURCES │ │
│ │ │ │ │ │
│ │ Text (code) │ │ Open file descriptors │ │
│ │ Data (globals) │ │ Network sockets │ │
│ │ Heap (malloc) │ │ Signal handlers │ │
│ │ Stack │ │ Current working directory │ │
│ │ mmap regions │ │ Environment variables │ │
│ └─────────────────┘ │ User/group IDs │ │
│ └──────────────────────────────────┘ │
│ │
│ KERNEL METADATA: │
│ PID (Process ID) │
│ PPID (Parent PID) │
│ State (Running/Sleeping/Zombie...) │
│ Priority / Nice value │
│ CPU affinity mask │
│ Cgroup memberships │
│ Namespace memberships │
└─────────────────────────────────────────────────────────────┘

Every process is described by a task_struct in the kernel — a C data structure with ~500 fields containing everything the kernel needs to know about a process.


3.2 Process Creation: fork(), exec(), clone()

Section titled “3.2 Process Creation: fork(), exec(), clone()”

fork() creates a copy of the calling process. After fork, you have two nearly identical processes: the parent and the child.

pid_t pid = fork();
if (pid == 0) {
// Child process
// pid == 0 means "I am the child"
} else if (pid > 0) {
// Parent process
// pid == child's PID
} else {
// Fork failed (errno set)
}
fork() — Copy-on-Write
──────────────────────
Before fork:
┌──────────────┐
│ Parent │
│ PID=100 │
│ memory: X │
└──────────────┘
After fork():
┌──────────────┐ ┌──────────────┐
│ Parent │ │ Child │
│ PID=100 │ │ PID=101 │
│ memory: X ─┼────┼─ memory: X │ ← Same physical pages!
└──────────────┘ └──────────────┘
When child writes to memory:
┌──────────────┐ ┌──────────────┐
│ Parent │ │ Child │
│ memory: X │ │ memory: X' │ ← New copy created for child
└──────────────┘ └──────────────┘
This is Copy-on-Write (COW) — no memory copied until needed.
Makes fork() very fast even for large processes.

exec() replaces the current process’s address space with a new program. It doesn’t create a new process — it transforms the current one.

Shell running a command: "ls -la /etc"
──────────────────────────────────────
1. bash calls fork()
→ Creates child bash process (PID=102)
2. Child bash calls exec("ls", ["-la", "/etc"])
→ Child's memory replaced with ls code
→ Child's name becomes "ls"
→ ls runs with PID=102
3. Parent bash calls wait(102)
→ Waits for ls to finish
4. ls exits, parent bash resumes
PID=1 (systemd)
└── PID=100 (bash) ← You are here
└── PID=102 (ls) ← After fork+exec

fork() and exec() are actually wrappers around the clone() syscall. clone() allows fine-grained control over what is shared between parent and child:

// fork() is roughly equivalent to:
clone(SIGCHLD, 0);
// Creating a thread is roughly:
clone(CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD, stack);
// Creating a container process:
clone(CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | CLONE_NEWUTS, 0);

This is one of the most commonly misunderstood concepts in systems programming.

Processes vs Threads in Linux
──────────────────────────────
MULTIPLE PROCESSES (each with own address space):
┌──────────────────┐ ┌──────────────────┐
│ Process A │ │ Process B │
│ PID = 100 │ │ PID = 101 │
│ │ │ │
│ ┌────────────┐ │ │ ┌────────────┐ │
│ │ Memory │ │ │ │ Memory │ │
│ │ (own copy)│ │ │ │ (own copy)│ │
│ └────────────┘ │ │ └────────────┘ │
│ ┌────────────┐ │ │ ┌────────────┐ │
│ │ FDs, etc │ │ │ │ FDs, etc │ │
│ └────────────┘ │ │ └────────────┘ │
└──────────────────┘ └──────────────────┘
Communication via IPC (pipes, sockets, shared mem)
Crash in A does NOT affect B
MULTIPLE THREADS (shared address space):
┌────────────────────────────────────────┐
│ Process A │
│ PID = 100 │
│ │
│ ┌──────────────────────────────────┐ │
│ │ Shared Memory │ │
│ │ (heap, globals, code) │ │
│ └──────────────────────────────────┘ │
│ │
│ ┌────────┐ ┌────────┐ ┌────────────┐ │
│ │Thread 1│ │Thread 2│ │ Thread 3 │ │
│ │ TID=100│ │ TID=102│ │ TID=103 │ │
│ │ Stack │ │ Stack │ │ Stack │ │
│ └────────┘ └────────┘ └────────────┘ │
└────────────────────────────────────────┘
Communication via shared memory (fast!)
Bug in Thread 1 can corrupt Thread 2's data
Thread crash often kills whole process

Key Linux fact: In Linux, threads and processes are both represented as tasks using task_struct. The difference is only in what they share. A thread is a process that shares its address space, file descriptors, and signal handlers with other threads.

Terminal window
# See threads of a process
ps -T -p <PID> # Show threads
# With top:
top -H # Toggle thread view
# Count threads
cat /proc/<PID>/status | grep Threads
# See all threads of nginx:
ps -T -p $(pidof nginx | awk '{print $1}')

A process is always in one of several states:

Process State Machine
─────────────────────
fork()
┌──────────────────────────────────────────────────────┐
│ CREATED │
└──────────────────────────┬───────────────────────────┘
│ Scheduler selects it
┌──────────────────────────────────────────────────────┐
│ RUNNING (R) │
│ Executing on a CPU core │
└────┬───────────────────┬──────────────────────────────┘
│ Wait for I/O │ Preempted by scheduler
▼ ▼
┌─────────────┐ ┌─────────────────────────────────┐
│ UNINTERRUPT │ │ RUNNABLE (R) │
│ SLEEP (D) │ │ Waiting for CPU (in run queue) │
│ │ └─────────────┬───────────────────┘
│ I/O pending │ │ Gets CPU time
└──────┬──────┘ └──────────────────────┐
│ I/O done │
└────────────────────────────────────────────────┤
┌──────────────────────────────────────────────────────┘
│ signal received while sleeping
┌─────────────────────────────────────────────────────┐
│ INTERRUPTIBLE SLEEP (S) │
│ Waiting for an event (I/O, timer, signal, lock) │
└──────────────────────────┬──────────────────────────┘
│ Event occurs or signal received
Back to RUNNABLE
exit() called:
┌─────────────────────────────────────────────────────┐
│ ZOMBIE (Z) │
│ Process exited, waiting for parent to read status │
│ Uses no resources except a PID and task_struct │
└──────────────────────────┬──────────────────────────┘
│ Parent calls wait()
TERMINATED (gone)
┌─────────────────────────────────────────────────────┐
│ STOPPED (T) │
│ Stopped by SIGSTOP or being debugged (ptrace) │
└─────────────────────────────────────────────────────┘
Terminal window
# See process states
ps aux
# STATE column: S=sleeping, R=running, D=uninterruptible sleep,
# Z=zombie, T=stopped, I=idle kernel thread
# Count processes by state
ps aux | awk '{print $8}' | sort | uniq -c
# Find processes in uninterruptible sleep (D state) - often I/O blocked
ps aux | grep " D"
# Find zombie processes
ps aux | grep " Z"
# A script to alert on too many D-state processes:
D_COUNT=$(ps aux | awk '$8 == "D" {count++} END {print count+0}')
if [ "$D_COUNT" -gt 5 ]; then
echo "ALERT: $D_COUNT processes in D state (possible I/O hang)"
fi

A process in D (uninterruptible sleep) cannot be killed — not even with kill -9. It is waiting for I/O that the kernel has committed to completing. Common causes:

  • NFS mount is unreachable
  • SCSI/NVMe device timeout
  • Hung system call
  • Kernel bug
Terminal window
# Diagnose D-state processes
# Find D-state processes
for pid in $(ps aux | awk '$8 == "D" {print $2}'); do
echo "=== PID $pid ==="
cat /proc/$pid/comm 2>/dev/null
cat /proc/$pid/wchan 2>/dev/null # What kernel function it's waiting in
cat /proc/$pid/stack 2>/dev/null # Kernel call stack
done

The Completely Fair Scheduler (CFS) is the default Linux process scheduler, introduced in kernel 2.6.23.

CFS’s goal is fairness: every runnable process should get an equal share of CPU time.

CFS Virtual Runtime Concept
────────────────────────────
Instead of traditional time slices, CFS tracks
"virtual runtime" (vruntime) per process.
vruntime = actual_runtime × (1 / weight)
(Lower weight processes accumulate vruntime faster)
CFS always runs the process with the LOWEST vruntime.
This ensures no process is starved.
Example: 3 processes with equal priority
─────────────────────────────────────────
Process A ran for 10ms: vruntime=10ms
Process B ran for 5ms: vruntime=5ms ← Lowest, runs next
Process C ran for 0ms: vruntime=0ms ← Actually lowest!
Time 0ms: [C runs] (vruntime=0)
Time 1ms: [B runs] (vruntime=5)
Time 6ms: [A runs] (vruntime=10)
Time 16ms: [C runs] (vruntime=1)
...
Over time, all processes get equal CPU time.

CFS stores all runnable processes in a red-black tree, ordered by vruntime. The leftmost node (lowest vruntime) is always the next process to run.

CFS Run Queue (Red-Black Tree)
──────────────────────────────
[P3: vrt=10]
/ \
[P1: vrt=5] [P5: vrt=20]
/ \
[P2: vrt=3] [P4: vrt=8]
Next to run: P2 (vrt=3) — leftmost node
O(log n) insertion and deletion
O(1) to find next process (cached pointer to leftmost)
Terminal window
# Nice values: -20 (highest priority) to +19 (lowest)
# Default nice = 0
# Run a command with lower priority (nice 10)
nice -n 10 backup_script.sh
# Renice a running process
renice -n 5 -p <PID>
# View processes sorted by CPU with nice values
ps aux --sort=-%cpu | head -20
# Real-time priorities (for time-critical processes)
# Real-time processes bypass CFS entirely
# SCHED_FIFO or SCHED_RR scheduler
# Priority 1-99 (99 = highest)
chrt -f 99 ./realtime_app # FIFO scheduling
chrt -r 50 ./realtime_app # Round-robin scheduling
# View scheduling policy of a process
chrt -p <PID>
Terminal window
# Load average: 1-min, 5-min, 15-min averages
cat /proc/loadavg
# Example: 2.50 1.80 1.20 3/456 12345
# │ │ │ │ │ └─ last PID created
# │ │ │ │ └──── total processes
# │ │ │ └─────── running/runnable
# │ │ └─────────── 15-min load
# │ └───────────────── 5-min load
# └─────────────────────── 1-min load
# Load average interpretation:
# Load = 1.0 on a 1-core system = 100% CPU used
# Load = 4.0 on a 4-core system = 100% CPU used
# Load = 4.0 on a 1-core system = 400% (4x overloaded)
# Formula: Ideal load = number of CPUs
CPUS=$(nproc)
LOAD=$(awk '{print $1}' /proc/loadavg)
echo "Load per CPU: $(echo "$LOAD / $CPUS" | bc -l)"
# Note: D-state processes ALSO count toward load average!
# A system with 50 D-state processes and no R-state processes
# can show high load but 0% CPU usage — I/O bottleneck

Terminal window
# The shell pipeline: cat file | grep pattern | sort | uniq
# Creates:
# bash (PID=100)
# └── cat (PID=101) ← fork+exec of cat
# └── grep (PID=102) ← fork+exec of grep (shares pipe with cat)
# └── sort (PID=103) ← fork+exec of sort
# └── uniq (PID=104) ← fork+exec of uniq
# See the process tree
pstree -p # Full process tree with PIDs
pstree -p $(pidof nginx) # Tree for nginx
# Using ps to see parent-child relationships
ps -ef --forest

A zombie is a process that has exited but whose parent hasn’t called wait() to collect its exit status.

Terminal window
# Find zombie processes
ps aux | awk '$8 == "Z" {print}'
# Zombies cannot be killed (they're already dead)
# You must kill the PARENT to clean up zombies
# Find parent of zombie:
ZOMBIE_PID=12345
PARENT_PID=$(ps -o ppid= -p $ZOMBIE_PID)
echo "Parent PID: $PARENT_PID"
# If parent is not responding, force-kill it
kill -KILL $PARENT_PID
# In production: many zombies = bug in the parent application
# (not calling wait() on child processes)

An orphan is a process whose parent has exited. Orphans are automatically adopted by PID 1 (systemd/init):

Parent (PID=100) dies unexpectedly
──────────────────────────────────
Before:
systemd (PID=1)
└── Parent (PID=100)
└── Child (PID=200)
After parent dies:
systemd (PID=1)
└── Child (PID=200) ← Adopted by init/systemd
Terminal window
# Check who the parent of a process is
ps -o pid,ppid,comm -p <PID>
# If PPID=1, the process has been orphaned and adopted by systemd

Signals are the primary mechanism for sending notifications between processes.

Common Signals
──────────────
Signal Number Default Action Description
──────────────────────────────────────────────────────
SIGHUP 1 Terminate Hangup (terminal disconnected)
Used to reload config
SIGINT 2 Terminate Interrupt (Ctrl+C)
SIGQUIT 3 Core dump Quit (Ctrl+\)
SIGKILL 9 Terminate Kill (cannot be caught/ignored!)
SIGUSR1 10 Terminate User-defined signal 1
SIGUSR2 12 Terminate User-defined signal 2
SIGTERM 15 Terminate Graceful termination request
SIGCHLD 17 Ignore Child stopped or exited
SIGSTOP 19 Stop Stop process (cannot be caught!)
SIGCONT 18 Continue Continue stopped process
Terminal window
# Send signals to processes
kill -TERM <PID> # Graceful shutdown (SIGTERM)
kill -HUP <PID> # Reload config (SIGHUP)
kill -KILL <PID> # Force kill (SIGKILL) — last resort
kill -USR1 <PID> # Application-specific (e.g., nginx reopen logs)
# Kill all processes by name
pkill -TERM nginx
# Send signal to entire process group
kill -TERM -<PGID>
# Production: Graceful nginx reload
kill -HUP $(cat /run/nginx.pid)
# or:
nginx -s reload
# Production: Force postgres checkpoint before maintenance
kill -CHECKPOINT $(head -1 /var/lib/postgresql/*/postmaster.pid)

Terminal window
# Step 1: Identify the problem
uptime
# 14:32:01 up 30 days, load average: 45.23, 42.10, 38.45
# Load >> number of CPUs → definitely saturated
nproc
# 16 cores → load of 45 means severely overloaded
# Step 2: Find what's consuming CPU
top -b -n1 | head -30
# Or: pidstat 1 5
# Step 3: Find hot functions within a process
perf top -p <PID>
# Step 4: Check if it's user code or kernel code
vmstat 1
# If %sy (system) is high → kernel issue
# If %us (user) is high → application issue
# Step 5: Check CFS scheduler stats per CPU
cat /proc/schedstat
# Or via perf:
perf stat -e context-switches,cpu-migrations sleep 5
Terminal window
# Process seems hung, consuming CPU but not making progress
# 1. Check what syscall it's stuck in
strace -p <PID>
# 2. Check kernel-level stack trace
cat /proc/<PID>/stack
# 3. If multiple threads, check all thread stacks
for tid in /proc/<PID>/task/*; do
echo "=== TID: $(basename $tid) ==="
cat $tid/stack 2>/dev/null
done
# 4. Check if it's holding a lock (in Go)
kill -QUIT <PID> # Go dumps goroutine stacks to stderr
# Or in Java:
kill -3 <PID> # Java dumps thread stacks

Q1: What is the difference between a process and a thread in Linux?

Answer: In Linux, both processes and threads are represented by the same kernel data structure (task_struct). The difference is in what they share: a process has its own address space (virtual memory), file descriptor table, and signal handlers. A thread shares all of these with other threads in the same process. Threads communicate via shared memory (fast) while processes communicate via IPC (pipes, sockets, shared memory segments — slower but safer). In Linux, fork() creates processes and pthread_create() (using clone() with sharing flags) creates threads.

Q2: What is the load average and what does it measure?

Answer: Load average is the average number of tasks in the run queue (running or waiting for CPU) plus tasks in uninterruptible sleep (D state, waiting for I/O) over the last 1, 5, and 15 minutes. A load of 1.0 represents full utilization of one CPU core. To interpret it, divide by the number of CPU cores: if you have 8 cores and load is 8.0, you’re at 100% utilization. Crucially, load does NOT measure CPU utilization — it counts both R-state (CPU-bound) and D-state (I/O-bound) processes. A system with all processes stuck waiting on I/O can show high load with 0% CPU usage.

Q3: What is a zombie process and how do you fix it?

Answer: A zombie is a process that has exited but remains in the process table because its parent hasn’t called wait() to retrieve its exit status. Zombies consume minimal resources (just a PID and a small task_struct), but accumulating many zombies can exhaust the PID namespace. You cannot kill a zombie directly — you must kill or signal the parent process so it calls wait(). If the parent is misbehaving, killing it will cause zombies to be adopted by PID 1 (systemd/init) which will reap them. In production, many zombie processes indicate a bug in the application’s child process management.

Q4: Why can’t you kill a process in D state with kill -9?

Answer: A process in D (uninterruptible sleep) state cannot receive signals, not even SIGKILL. It is blocked deep inside a kernel I/O path where it’s unsafe to be interrupted — the kernel has committed to completing the I/O operation. This is by design: interrupting certain I/O operations mid-way could corrupt data structures. Common causes include: hung NFS mounts, storage device timeouts, or buggy device drivers. The only solutions are to fix the underlying I/O problem (e.g., reconnect the NFS server), wait for a kernel timeout, or reboot.


ConceptKey Point
ProcessProgram instance with own address space, PID, resources
ThreadLightweight process sharing address space
fork()Creates child as copy of parent (COW)
exec()Replaces current process with new program
CFSCompletely Fair Scheduler using virtual runtime
D stateUninterruptible sleep — cannot be killed
ZombieExited but parent hasn’t called wait()
Load averageCounts R + D state tasks, not just CPU usage

Chapter 1 (Linux Architecture), Chapter 2 (Boot Process) — know what user space and kernel space mean.


Advantages: Efficient process creation via copy-on-write (fork), robust isolation. Disadvantages: Zombie processes if parents don’t wait(), context switching adds overhead under heavy load.


  • Using kill -9 (SIGKILL) as a first resort — always try SIGTERM first to allow graceful cleanup of connections and temp files.
  • Confusing PID and TID: threads share a PID but each has a unique TID; ps -T shows threads.
  • Assuming fork() immediately copies all parent memory — Linux uses Copy-on-Write; pages only copied on write.
  • Not understanding zombie processes: a process that exited but whose parent hasn’t called wait() — accumulate until parent exits.
  • Reading top’s CPU% as absolute: 100% on 8 cores = 12.5% total. Use mpstat -P ALL for per-core view.

SymptomCauseDiagnosisFix
Process stuck in D state foreverBlocked on hung I/O (NFS, disk)`ps auxgrep ’ D ’; strace -p PID`
Zombie process accumulationParent not calling wait() on dead children`ps auxgrep ’ Z ’; find parent: ps -o ppid= -p ZOMBIE_PID`
Fork bomb / pid table exhaustedRunaway recursive fork()`watch ‘cat /proc/sys/kernel/pid_max’; top -b -n1head`
Process not responding to SIGTERMApplication ignoring SIGTERM signalstrace -e signal -p PIDFix app to handle SIGTERM; use SIGKILL as last resort after timeout

Objective: Observe the full process lifecycle live.

Terminal window
# 1. View process tree
pstree -p | head -30
# 2. Trace fork/exec for a command
strace -e trace=fork,clone,execve ls /tmp 2>&1
# 3. Monitor process states
watch -n 1 'ps aux | awk "{print \$8}" | sort | uniq -c'
# 4. Create background process and inspect
sleep 300 &
BG=$!
echo "PID: $BG"
cat /proc/$BG/status | grep -E "Name|State|Pid|PPid|Threads"
kill -SIGTERM $BG
# 5. Scheduler stats for this shell
cat /proc/$$/sched | head -15
# 6. CPU affinity of current process
taskset -p $$
# 7. Change process priority
nice -n 10 sleep 60 &
ps -p $! -o pid,ni,pri,stat,cmd

  1. Create a zombie process in C: fork a child that exits immediately, but have the parent sleep for 60s without calling wait(). Observe it with ps aux | grep Z.
  2. Write a shell script that traps SIGTERM and logs ‘Graceful shutdown started’ before exiting. Test it with kill -SIGTERM PID.
  3. Use taskset -c 0 stress-ng --cpu 1 to pin CPU stress to core 0. Verify with htop showing only core 0 loaded.
  4. Find the process with the longest running time on your system using ps aux --sort=-etime | head -5.
  5. Set nice -n 19 (lowest priority) on a CPU-intensive process and measure how it affects its completion time vs nice -n -20.

  • Process lifecycle: fork() → exec() → run → exit() → wait() by parent → PID recycled.
  • States: R=running, S=sleeping interruptible, D=uninterruptible I/O wait, Z=zombie, T=stopped.
  • Every process inherits: UID/GID, open FDs, signal handlers, environment variables, working directory from parent.
  • Copy-on-Write: fork() is cheap because pages are shared until written — then only the written page is copied.
  • SIGTERM (15) = please stop gracefully. SIGKILL (9) = die immediately (cannot be caught). SIGSTOP (19) = pause.
  • CFS (Completely Fair Scheduler) tracks vruntime — always schedules the process with lowest accumulated CPU time.


  • Chapter 1 — Kernel and user space overview
  • Chapter 4 — Process virtual memory layout
  • Chapter 5 — Process resource isolation
  • Chapter 6 — Signals and inter-process communication

Last Updated: July 2026