Skip to content
Flowchart Learning

Kernel and Shell

Chapter 4: Linux Kernel and Shell

Section titled “Chapter 4: Linux Kernel and Shell”

Overview

Section titled “Overview”

This chapter covers the Linux kernel architecture, kernel modules, and the shell - the primary interface for interacting with the system.

Why This Matters in DevOps/SRE

Section titled “Why This Matters in DevOps/SRE”

Kernel and shell knowledge is essential for automation, debugging, and system optimization:

                 Kernel & Shell for DevOps/SRE
+------------------------------------------------------------------+
|                                                                   |
|    Automation & Scripting:                                         |
|    +----------------------------------------------------------+  |
|    | 90% of DevOps tasks are automated via shell scripts    |  |
|    | CI/CD pipelines run bash scripts                       |  |
|    | Infrastructure as Code uses shell under the hood       |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Debugging & Troubleshooting:                                    |
|    +----------------------------------------------------------+  |
|    | strace/ltrace → System call tracing                    |  |
|    | /proc filesystem → Runtime kernel info                  |  |
|    | lsmod → Loaded kernel modules                          |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Performance Optimization:                                       |
|    +----------------------------------------------------------+  |
|    | sysctl → Kernel parameter tuning                       |  |
|    | tune2fs → Filesystem optimization                      |  |
|    | ip route → Network stack tuning                       |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+

Practical Impact:

  • Write efficient automation scripts for deployment pipelines
  • Debug application issues using strace, lsof, /proc
  • Tune kernel parameters for performance
  • Troubleshoot container networking (namespaces, cgroups)

4.1 Linux Kernel Architecture

Section titled “4.1 Linux Kernel Architecture”

Kernel Components

Section titled “Kernel Components”

The Linux kernel is a monolithic kernel with loadable modules.

+------------------------------------------------------------------+
|                        Linux System Architecture                   |
+------------------------------------------------------------------+


                        +-----------------------+
                        |     User Space        |
+-----------------------+-----------------------+--------------------+
|    Applications       |    Libraries (glibc)  |    Shell (bash)   |
+-----------------------+-----------------------+--------------------+


+-------------------------------------------------------------------+
|                        Kernel Space                                |
+-----------------------+-----------------------+--------------------+
|   System Call Interface|    Kernel Subsystems |   Device Drivers |
+-----------------------+-----------------------+--------------------+


+-------------------------------------------------------------------+
|                         Hardware                                   |
+-----------------------+-----------------------+--------------------+
|   CPU                 |   Memory (RAM)       |   Disk/NIC        |
+-----------------------+-----------------------+--------------------+

Kernel Subsystems

Section titled “Kernel Subsystems”
+------------------------------------------------------------------+
|                      Kernel Subsystems                            |
+------------------------------------------------------------------+


    +----------------+     +----------------+     +----------------+
    | Process        |     | Memory         |     | File System    |
    | Scheduler      |     | Manager        |     | (VFS)          |
    +----------------+     +----------------+     +----------------+
           |                      |                     |
           +----------------------+---------------------+
                                  |
                         +----------------+
                         | Network Stack  |
                         +----------------+
                                  |
                         +----------------+
                         | IPC            |
                         +----------------+


+------------------------------------------------------------------+
| Subsystem      | Description              | Key Files             |
+----------------+-------------------------+-----------------------+
| Process Sched | Manages CPU time         | kernel/sched/*        |
| Memory Manager| Virtual memory, paging   | mm/*.c                |
| Virtual FS    | Abstraction for FS       | fs/*.c                |
| Network Stack | TCP/IP, sockets          | net/*.c               |
| IPC           | Inter-process comm       | ipc/*.c               |
| Device Drivers| Hardware support         | drivers/*             |
+----------------+-------------------------+-----------------------+

4.2 Kernel Modules

Section titled “4.2 Kernel Modules”

Working with Modules

Section titled “Working with Modules”
Terminal window
# List loaded modules
lsmod


# Load a module
modprobe module_name


# Unload a module
modprobe -r module_name


# Show module information
modinfo module_name


# Load module with options
modprobe module_name parameter=value

Module Configuration

Section titled “Module Configuration”
+------------------------------------------------------------------+
|                    Module Management                              |
+------------------------------------------------------------------+


  Load Module     +--------+     Remove Module
  --------------> |  mod   | <--------------
  modprobe name   |probe   |   modprobe -r name
                  +--------+


                  Module Info
                  +--------+
                  | modinfo|
                  +--------+
Terminal window
# Blacklist a module (prevent loading)
echo "blacklist module_name" > /etc/modprobe.d/blacklist.conf


# Module dependencies
depmod -a


# View module parameters
cat /sys/module/module_name/parameters/parameter_name


# Set module parameter
echo value > /sys/module/module_name/parameters/parameter_name

4.3 The Shell

Section titled “4.3 The Shell”

Common Shells

Section titled “Common Shells”
+------------------------------------------------------------------+
|                      Shell Types                                  |
+------------------------------------------------------------------+


    +----------+     +----------+     +----------+     +----------+
    |   bash   |     |   zsh    |     |   fish   |     |   dash   |
    +----------+     +----------+     +----------+     +----------+
        |                 |                 |                 |
        v                 v                 v                 v
    Default on       Power user       Beginner-        Fast, minimal
    most Linux       features          friendly          scripts
    systems


+------------------------------------------------------------------+
| Shell    | Config File        | Purpose           | Best For    |
+----------+--------------------+-------------------+--------------+
| bash     | .bashrc, .bash_    | Default shell     | General use  |
|          | profile            |                   |              |
| zsh      | .zshrc            | Extended features | Power users  |
| fish     | config.fish       | User-friendly     | Beginners    |
| dash     | (no interactive)  | Fast, lightweight | Scripts      |
+----------+--------------------+-------------------+--------------+

Starting a Shell

Section titled “Starting a Shell”
Terminal window
# Check available shells
cat /etc/shells


# Check current shell
echo $SHELL
echo $0


# Change shell temporarily
bash
zsh
fish


# Change default shell permanently
chsh -s /bin/zsh

4.4 Shell Configuration

Section titled “4.4 Shell Configuration”

Shell Configuration Files

Section titled “Shell Configuration Files”
+------------------------------------------------------------------+
|                 Shell Configuration Files                         |
+------------------------------------------------------------------+


    System-wide              User-specific
    +-----------+            +-------------+
    | /etc/     |            | ~/.bash_    |
    | profile   |            | profile     |
    +-----------+            +-------------+
          |                       |
          v                       v
    +-----------+            +-------------+
    | /etc/bash.|            | ~/.bashrc   |
    | bashrc    |            +-------------+
    +-----------+                  |
                                  v
                            +-------------+
                            | ~/.bash_    |
                            | history     |
                            +-------------+
FileScopeWhen Loaded
/etc/profileSystem-wideLogin shells
/etc/bash.bashrcSystem-wideInteractive bash
~/.bash_profileUserLogin shells
~/.bashrcUserInteractive non-login
~/.bash_historyUserCommand history
Terminal window
# Common .bashrc additions
alias ll='ls -la'
alias la='ls -a'
export EDITOR=vim
export PATH=$PATH:/custom/path

Environment Variables

Section titled “Environment Variables”
+------------------------------------------------------------------+
|                 Environment Variables Flow                        |
+------------------------------------------------------------------+


  System Default      +----------+     User Override
  /etc/environment   | export   |    ~/.bashrc
  -----------------> | VAR=val  | <---------------
                     +----------+


                     +----------+
                     | printenv |
                     +----------+
                          |
                          v
                     +----------+
                     | $VAR     |
                     +----------+
Terminal window
# View all environment variables
env
printenv


# View specific variable
echo $HOME
printenv HOME


# Set variable (current shell only)
export VAR=value


# Set variable permanently (add to ~/.bashrc)
echo 'export VAR=value' >> ~/.bashrc

Important Variables

Section titled “Important Variables”
VariableDescription
HOMEUser’s home directory
PATHDirectories to search for commands
USERCurrent username
PWDCurrent working directory
SHELLDefault shell
EDITORDefault text editor
LANGLanguage and locale settings

4.5 Shell Scripting Basics

Section titled “4.5 Shell Scripting Basics”

Shebang

Section titled “Shebang”
#!/bin/bash
#!/bin/sh
#!/usr/bin/env bash

Variables

Section titled “Variables”
+------------------------------------------------------------------+
|                    Variable Types                                  |
+------------------------------------------------------------------+


  String              Array
  +-------+          +-------+
  | name= |          | arr=( |
  | "John"|          | one   |
  +-------+          | two   |
                     | three |
                     | )     |
                     +-------+
                       |
                       v
                  +-------+
                  | ${arr}|
                  +-------+
Terminal window
# Variable assignment (no spaces around =)
name="John"
age=30


# Read-only variable
readonly CONSTANT="value"


# Array
array=(one two three)
echo ${array[0]}
echo ${array[@]}

Control Structures

Section titled “Control Structures”
+------------------------------------------------------------------+
|                    Control Flow                                   |
+------------------------------------------------------------------+


  +--------+     +--------+     +--------+     +--------+
  |   if   |---->|  elif  |---->|  else  |---->|   fi   |
  +--------+     +--------+     +--------+     +--------+


  +--------+     +--------+     +--------+
  |  for   |---->|  done  |---->|   esac  |
  +--------+     +--------+     +--------+


  +--------+     +--------+     +--------+
  |  while |---->|  done  |---->|   esac  |
  +--------+     +--------+     +--------+
Terminal window
# If statement
if [ condition ]; then
    command
elif [ condition ]; then
    command
else
    command
fi


# Case statement
case $var in
    pattern1)
        command
        ;;
    pattern2)
        command
        ;;
    *)
        default
        ;;
esac


# For loop
for i in {1..5}; do
    echo $i
done


# While loop
while read line; do
    echo $line
done < file.txt

Functions

Section titled “Functions”
+------------------------------------------------------------------+
|                    Function Structure                             |
+------------------------------------------------------------------+


  Function Call              Function Definition
  +----------+              +--------------------------+
  | func arg | ------------>| function_name() {       |
  +----------+              |   local arg1=$1         |
                           |   local arg2=$2         |
                           |   echo "..."            |
                           |   return 0              |
                           | }                       |
                           +--------------------------+
Terminal window
function_name() {
    local arg1=$1
    local arg2=$2
    echo "Arguments: $arg1 $arg2"
    return 0
}


# Call function
function_name val1 val2

4.6 Input/Output

Section titled “4.6 Input/Output”

Redirection

Section titled “Redirection”
+------------------------------------------------------------------+
|                    I/O Redirection                                |
+------------------------------------------------------------------+


  Standard Input (stdin)   Standard Output (stdout)  Standard Error (stderr)
         |                         |                        |
         v                         v                        v
      +-------+                +-------+                +-------+
      |   0   |                |   1   |                |   2   |
      +-------+                +-------+                +-------+


  Redirection Operators:


  >  file     - Redirect stdout (overwrite)
  >> file     - Redirect stdout (append)
  2> file     - Redirect stderr
  &> file     - Redirect both
  <  file     - Redirect stdin


  command > output.txt    - Save output to file
  command 2> error.txt   - Save errors to file
  command > out.txt 2>&1 - Save both to same file
Terminal window
# Standard output to file (overwrite)
command > output.txt


# Standard output to file (append)
command >> output.txt


# Standard error to file
command 2> error.txt


# Redirect both stdout and stderr
command > output.txt 2>&1
command &> output.txt


# Redirect stdin from file
command < input.txt


# Here document
cat << EOF
Multi-line
text here
EOF


# Here string
command <<< "input string"

Pipes

Section titled “Pipes”
+------------------------------------------------------------------+
|                    Pipeline                                       |
+------------------------------------------------------------------+


  +--------+     +--------+     +--------+     +--------+
  | Input  |---->| cmd1   |---->| cmd2   |---->| Output |
  +--------+     +--------+     +--------+     +--------+
                      |              |
                      v              v
                  +--------+    +--------+
                  | stdout |    | stdin  |
                  +--------+    +--------+
Terminal window
# Pipe output to another command
command1 | command2


# Pipeline examples
ls -l | grep "pattern"
cat file | sort | uniq
head -n 10 file | tail -n 5

4.7 Process Management

Section titled “4.7 Process Management”

Viewing Processes

Section titled “Viewing Processes”
+------------------------------------------------------------------+
|                    Process Hierarchy                               |
+------------------------------------------------------------------+


                       systemd (PID 1)
                              |
         +--------+-----------+--------+--------+
         |        |           |        |        |
         v        v           v        v        v
      sshd     httpd       mysqld   docker   nginx
         |                                   |
         v                                   v
       ssh                               php-fpm
                                          |
                                          v
                                       (children)
Terminal window
# Process status
ps
ps aux
ps -ef


# Top processes
top
htop
atop


# Specific process
pgrep process_name
pkill process_name

Job Control

Section titled “Job Control”
+------------------------------------------------------------------+
|                    Job Control                                    |
+------------------------------------------------------------------+


  Foreground              Background              Jobs List
  +--------+             +--------+             +--------+
  | Ctrl+Z |             |   &    |             | jobs   |
  +--------+             +--------+             +--------+
       |                    |                      |
       v                    v                      v
  +--------+             +--------+             +--------+
  | Stop   |             | Run    |             | fg %1  |
  | process|             | in bg  |             | bg %2   |
  +--------+             +--------+             +--------+
Terminal window
# Run in background
command &


# View jobs
jobs


# Bring to foreground
fg %job_number


# Send to background (from foreground)
Ctrl+Z
bg %job_number


# Kill job
kill %job_number

4.8 Shell Expansion

Section titled “4.8 Shell Expansion”

Types of Expansion

Section titled “Types of Expansion”
+------------------------------------------------------------------+
|                    Shell Expansion Types                           |
+------------------------------------------------------------------+


  Brace Expansion      Tilde Expansion       Command Substitution
  +----------+         +----------+         +----------+
  | file{1..|         | ~        |         | $(date)  |
  | 3}.txt  |         | ~user   |         | `date`   |
  +----------+         +----------+         +----------+
        |                   |                   |
        v                   v                   v
  file1.txt,          /home/user          Sat Mar  8...
  file2.txt,
  file3.txt


  Arithmetic Expansion      Parameter Expansion
  +----------+              +---------------+
  | $((2+3)) |              | ${var:-default|
  +----------+              | ${var#pattern}|
                            +---------------+
Terminal window
# Brace expansion
echo {a,b,c}        # a b c
echo file{1..3}.txt   # file1.txt file2.txt


# Tilde expansion
echo ~              # /home/user
echo ~user         # /home/user


# Command substitution
echo $(date)
echo `date`


# Arithmetic expansion
echo $((2 + 3))    # 5
echo $((10 / 2))   # 5

Parameter Expansion

Section titled “Parameter Expansion”
Terminal window
# Default value
${var:-default}    # Use default if unset
${var:=default}   # Set default if unset


# String operations
${var#pattern}     # Remove shortest match from beginning
${var##pattern}   # Remove longest match from beginning
${var%pattern}    # Remove shortest match from end
${var%%pattern}   # Remove longest match from end
${var/old/new}    # Replace first occurrence
${var//old/new}   # Replace all occurrences
${#var}           # String length

4.9 Troubleshooting

Section titled “4.9 Troubleshooting”

Common Shell Issues

Section titled “Common Shell Issues”
+------------------------------------------------------------------+
|                    Troubleshooting Guide                            |
+------------------------------------------------------------------+


  Problem                    Solution
  +----------------------+  +--------------------------------+
  | "Command not found" |  | Check PATH: echo $PATH        |
  +----------------------+  | Add: export PATH=$PATH:/path  |
                           +--------------------------------+


  +----------------------+  +--------------------------------+
  | "Permission denied" |  | Check: ls -l file             |
  +----------------------+  | Fix: chmod +x script.sh      |
                           +--------------------------------+


  +----------------------+  +--------------------------------+
  | Variable not set    |  | Check: ${var:-"default"}     |
  +----------------------+  | Test: if [ -z "$VAR" ]...    |
                           +--------------------------------+


  +----------------------+  +--------------------------------+
  | Quote issues        |  | Use " " for expansion        |
  +----------------------+  | Use ' ' for literal          |
                           +--------------------------------+

  1. Command not found

    Terminal window
    # Check PATH
    echo $PATH
    # Add to PATH
    export PATH=$PATH:/new/path

  2. Permission denied

    Terminal window
    # Check file permissions
    ls -l filename
    # Add execute permission
    chmod +x script.sh

  3. Variable not set

    Terminal window
    # Use default value
    echo ${VAR:-"default"}
    # Check if set
    if [ -z "${VAR}" ]; then echo "VAR is not set"; fi

  4. Quotes issues

    Terminal window
    # Double quotes allow expansion
    echo "Home: $HOME"
    # Single quotes preserve literally
    echo 'Home: $HOME'  # Output: Home: $HOME

4.10 Practice Exercises

Section titled “4.10 Practice Exercises”
  1. Write a script that accepts a username and displays their information
  2. Create a function that calculates the factorial of a number
  3. Write a script that monitors a specific process and alerts if it stops
  4. Create a backup script that tar-gzips specified directories
  5. Write a script that parses a log file and extracts error messages

Common Mistakes & Anti-Patterns

Section titled “Common Mistakes & Anti-Patterns”

1. Not Using Shell Expansion Properly

Section titled “1. Not Using Shell Expansion Properly”
Terminal window
# ❌ WRONG: Not understanding expansion order
rm -rf $VAR/*    # If VAR is empty, deletes everything!
rm -rf ${VAR}/*  # Safer but still risky


# ✅ CORRECT: Always check variables first
echo "$VAR"
[ -n "$VAR" ] && rm -rf "$VAR"/*
rm -rf -- "$VAR"/*  # Use -- to separate options from arguments

2. Ignoring Kernel Module Dependencies

Section titled “2. Ignoring Kernel Module Dependencies”
Terminal window
# ❌ WRONG: Loading modules without checking dependencies
modprobe nf_conntrack  # May fail if dependent modules not loaded


# ✅ CORRECT: Check dependencies first
modprobe --show-depends nf_conntrack
lsmod                 # List loaded modules

3. Improper Script Error Handling

Section titled “3. Improper Script Error Handling”
# ❌ WRONG: Not checking exit codes
#!/bin/bash
cd /some/directory
rm -rf *  # If cd fails, still runs rm in wrong directory!


# ✅ CORRECT: Use set -e and check errors
#!/bin/bash
set -euo pipefail
cd /some/directory || exit 1
rm -rf *

4. Not Using sysctl Properly

Section titled “4. Not Using sysctl Properly”
Terminal window
# ❌ WRONG: Setting kernel parameters without persistence
sysctl -w net.ipv4.ip_forward=1  # Lost after reboot!


# ✅ CORRECT: Add to /etc/sysctl.conf
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p  # Apply all settings from config

Interview Questions

Section titled “Interview Questions”
  1. What is the difference between a system call and a library function?
  2. How do kernel modules work and when would you need to load/unload them?
  3. Explain the difference between bash and sh
  4. What is the purpose of /proc filesystem?
  5. How does shell expansion work? Explain the order.
  6. What are the differences between hard links and symbolic links?
  7. How would you debug a script that’s behaving unexpectedly?
  8. What is the purpose of sysctl and how do you make changes persistent?

Quick Reference

Section titled “Quick Reference”
+------------------------------------------------------------------+
|                    Quick Reference                                 |
+------------------------------------------------------------------+


  File Operations:
  +----------------------------------------------------------+
  | ls -la          | List all files with details            |
  | cd              | Change directory                       |
  | mkdir -p        | Create directory tree                  |
  | rm -rf          | Force remove directory                 |
  | cp -r           | Copy recursively                       |
  | mv              | Move/rename                            |
  +----------------------------------------------------------+


  Process Management:
  +----------------------------------------------------------+
  | ps aux         | Show all processes                     |
  | top            | Interactive process viewer             |
  | kill           | Send signal to process                 |
  | bg/fg         | Background/foreground jobs              |
  +----------------------------------------------------------+


  Text Processing:
  +----------------------------------------------------------+
  | grep           | Search for pattern                     |
  | sed            | Stream editor                           |
  | awk            | Text processing                         |
  | sort/uniq      | Sort and remove duplicates              |
  +----------------------------------------------------------+