Filesystem Internals: VFS, OverlayFS, ext4, XFS

Chapter 8: Filesystem Internals: VFS, OverlayFS, ext4, XFS

Learning Objectives

Understand the VFS layer and how it abstracts different filesystems
Understand the ext4 and XFS filesystem internals
Understand OverlayFS and how container layers work
Know filesystem tuning and monitoring for production

Prerequisites

What is this? (Beginner On-Ramp)

A filesystem is how data is organized and stored on a physical disk. Without it, a hard drive is just a massive, unorganized bucket of 1s and 0s. The filesystem provides the structure of folders and files, keeping track of where exactly on the disk each piece of data lives.

8.1 The VFS: Virtual File System

    VFS Architecture
    ─────────────────

    Applications: open(), read(), write(), stat(), mkdir()
    ─────────────────────────────────────────────────────
                          ↓
    ┌─────────────────────────────────────────────────────────┐
    │              VFS (Virtual File System)                   │
    │                                                          │
    │  Key Objects:                                           │
    │  • superblock  — mounted filesystem metadata            │
    │  • inode       — file/directory metadata                │
    │  • dentry      — directory entry (name → inode mapping) │
    │  • file        — open file instance (per fd)            │
    │                                                          │
    │  Key Caches:                                            │
    │  • dcache      — directory entry cache (name lookups)   │
    │  • icache      — inode cache (file metadata)            │
    │  • page cache  — file data cache                        │
    └───────────┬──────────────────────────────┬──────────────┘
                │                              │
                ▼                              ▼
    ┌─────────────────┐              ┌─────────────────┐
    │     ext4        │              │      XFS        │
    └─────────────────┘              └─────────────────┘
                │                              │
    ┌─────────────────┐              ┌─────────────────┐
    │     tmpfs       │              │      NFS        │
    └─────────────────┘              └─────────────────┘
                │                              │
    ┌─────────────────┐              ┌─────────────────┐
    │    OverlayFS    │              │      Btrfs      │
    └─────────────────┘              └─────────────────┘

Key VFS Objects

inode: Stores file metadata (permissions, timestamps, size, disk block pointers). Does NOT store the filename.

dentry: Maps a filename to an inode. The directory entry /etc/nginx/nginx.conf is a dentry pointing to the nginx.conf inode.

Why the separation?: Hard links. Multiple dentries can point to the same inode. The inode’s link count tracks how many dentries reference it.

# See inode information
stat /etc/nginx/nginx.conf
# Size: 1234       Blocks: 8          IO Block: 4096
# Device: fd01h    Inode: 1234567     Links: 1
# Access: -rw-r--r--  Uid: (0/root)   Gid: (0/root)
# Access: 2024-01-15 10:00:00
# Modify: 2024-01-10 09:00:00
# Change: 2024-01-10 09:00:00

# Get just the inode number
ls -i /etc/nginx/nginx.conf

# Create a hard link (same inode, different dentry)
ln /etc/nginx/nginx.conf /tmp/nginx-conf-link
stat /tmp/nginx-conf-link
# Same inode number, Links: 2

# Find all hard links to an inode
find / -inum 1234567 2>/dev/null

8.2 ext4: The Workhorse Filesystem

ext4 (Fourth Extended Filesystem) is the default filesystem for most Linux distributions.

ext4 Layout

    ext4 Disk Layout
    ─────────────────
    ┌────────────────────────────────────────────────────────┐
    │  Block 0: Superblock (copies throughout disk)          │
    │  - Total blocks, block size, inode count               │
    │  - Mount count, last mount time                        │
    │  - UUID, filesystem name                               │
    ├────────────────────────────────────────────────────────┤
    │  Block Group 0:                                        │
    │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌───────┐ │
    │  │  GDT     │  │  Block   │  │  Inode   │  │ Inode │ │
    │  │  (Group  │  │  Bitmap  │  │  Bitmap  │  │ Table │ │
    │  │  Descr.) │  │          │  │          │  │       │ │
    │  └──────────┘  └──────────┘  └──────────┘  └───────┘ │
    │  ┌────────────────────────────────────────────────┐   │
    │  │                Data Blocks                      │   │
    │  └────────────────────────────────────────────────┘   │
    ├────────────────────────────────────────────────────────┤
    │  Block Group 1: (same structure)                       │
    │  ...                                                   │
    └────────────────────────────────────────────────────────┘

ext4 Key Features

Extents: ext4 uses extents instead of block lists. An extent is a contiguous range of blocks, described as (start_block, length). Dramatically reduces fragmentation for large files.

Journal (Write-Ahead Log): Before modifying filesystem metadata, ext4 writes the changes to a journal first. If the system crashes mid-write, the journal lets fsck recover quickly without a full filesystem scan.

# ext4 filesystem information
tune2fs -l /dev/sda1
# Filesystem volume name:   production-data
# Block count:              10240000
# Block size:               4096
# Inode count:              2621440
# Journal size:             128M
# Features:                 has_journal ext_attr resize_inode

# Check and repair ext4
fsck.ext4 -n /dev/sda1    # Check only (no changes)
fsck.ext4 /dev/sda1       # Fix (unmount first!)

# Tune ext4 performance
# Disable access time updates (saves I/O on every read)
mount -o remount,noatime /dev/sda1 /

# Or in /etc/fstab:
# UUID=... / ext4 defaults,noatime,nodiratime 0 1

# View ext4 statistics
dumpe2fs -h /dev/sda1 | grep -E "Free blocks|Free inodes|Block size"

Inode Exhaustion

# Check inode usage (can be full even with free blocks!)
df -i /
# Filesystem      Inodes  IUsed    IFree IUse% Mounted on
# /dev/sda1      2621440  2621439      1  100% /
# ALERT: 100% inode usage! Can't create new files!

# Find directories with most files (likely the culprit)
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n -r | head -10

# Common cause: many small temp files, logs, email queue
# Fix: delete temp files or increase inode count (requires mkfs)

# When creating a filesystem, set inode count based on expected file count
mkfs.ext4 -i 4096 /dev/sdb1   # One inode per 4KB (more inodes, smaller files)
mkfs.ext4 -i 65536 /dev/sdb1  # One inode per 64KB (fewer inodes, large files)

8.3 XFS: High-Performance Filesystem

XFS is preferred for large files, high-throughput workloads, and large filesystems (> 1TB). Used by RHEL/CentOS as default since version 7.

    XFS vs ext4 Comparison
    ─────────────────────────────────────────────────────────────
    Feature          XFS                    ext4
    ─────────────────────────────────────────────────────────────
    Max FS size      8 exabytes             1 exabyte
    Max file size    8 exabytes             16TB
    Default inode    Dynamic (no exhaustion) Fixed at mkfs time
    Parallel I/O     Excellent (AG-based)   Good
    Small files      Adequate               Excellent
    Defragmentation  Online xfs_fsr         Offline e4defrag
    Delayed alloc    Yes                    Yes
    Journaling       Metadata only          Metadata + data
    Shrink FS        No (XFS limitation!)   Yes (offline)
    ─────────────────────────────────────────────────────────────

# XFS filesystem information
xfs_info /dev/sda1

# Check and repair XFS
xfs_repair /dev/sda1        # Repair (must be unmounted)
xfs_repair -n /dev/sda1     # Check only

# XFS tuning
# Set log stripe size (performance for RAID)
mkfs.xfs -d su=64k,sw=4 /dev/md0

# Freeze filesystem (for consistent snapshots)
xfs_freeze -f /mountpoint   # Freeze I/O
# Take snapshot here
xfs_freeze -u /mountpoint   # Unfreeze

# XFS performance stats
xfs_stats /proc/fs/xfs/stat

8.4 OverlayFS: How Container Layers Work

OverlayFS is the foundation of Docker container images. It allows multiple filesystem layers to be stacked efficiently.

    OverlayFS Layer Model
    ──────────────────────

    When you build a Docker image:

    ┌────────────────────────────────────────────────────────┐
    │         Container Layer (Read-Write)                    │
    │  /app/config.json (modified)                           │
    │  /tmp/cache (new file)                                 │
    └────────────────────────────────────────────────────────┘
                            ↕  merged view (union)
    ┌────────────────────────────────────────────────────────┐
    │         Image Layer 3 - App Layer (Read-Only)           │
    │  FROM node:18 → COPY ./app /app                        │
    └────────────────────────────────────────────────────────┘
                            ↕
    ┌────────────────────────────────────────────────────────┐
    │         Image Layer 2 - Deps Layer (Read-Only)          │
    │  RUN npm install                                       │
    └────────────────────────────────────────────────────────┘
                            ↕
    ┌────────────────────────────────────────────────────────┐
    │         Image Layer 1 - Base OS (Read-Only)             │
    │  FROM ubuntu:22.04                                     │
    └────────────────────────────────────────────────────────┘

    When container reads a file:
    Container Layer → Layer 3 → Layer 2 → Layer 1
    Returns first match found (top-down)

    When container writes to a file from a lower layer:
    Copy-on-Write: file is copied to Container Layer first,
    then modified there. Lower layers remain unchanged.

# OverlayFS mount (manual)
mkdir upper lower merged work

# Create lower (read-only) layer
echo "from lower" > lower/test.txt

# Mount overlay
mount -t overlay overlay \
    -o lowerdir=lower,upperdir=upper,workdir=work \
    merged

# Read the file (comes from lower layer)
cat merged/test.txt   # "from lower"

# Modify it (triggers copy-on-write to upper)
echo "modified" > merged/test.txt

# Lower unchanged, upper has modified copy
cat lower/test.txt    # "from lower"
cat upper/test.txt    # "modified"

# Clean up
umount merged

# In Docker:
# See OverlayFS mounts
docker inspect <container> | grep -A10 "GraphDriver"
# Shows: LowerDir, UpperDir, MergedDir, WorkDir

OverlayFS Performance

# OverlayFS can have performance issues with many layers
# Each file lookup traverses all layers

# Check how many layers a Docker image has
docker history <image> | wc -l

# Reduce layers with multi-stage builds
# and combining RUN commands:
# Bad: 10 separate RUN layers
# Good: 1 RUN layer with && between commands

# View actual overlay mounts
cat /proc/mounts | grep overlay

# Check overlay disk usage
du -sh /var/lib/docker/overlay2/

8.5 Filesystem Mount Options for Production

# /etc/fstab production tuning

# For root filesystem (OS):
UUID=... / ext4 defaults,noatime,nodiratime,errors=remount-ro 0 1
# noatime: don't update access time on every read (big perf win)
# errors=remount-ro: if error, remount read-only instead of panic

# For data directory (database, logs):
UUID=... /data xfs noatime,nodiratime,allocsize=64m 0 2
# allocsize: hint for preallocation (reduces fragmentation)

# For tmpfs (in-memory filesystem):
tmpfs /run/myapp tmpfs defaults,size=512m,mode=0750 0 0

# Check mount options of a filesystem
cat /proc/mounts | grep /data
findmnt /data

# Remount with new options (without unmounting)
mount -o remount,noatime /data

8.6 Filesystem Monitoring

# ── Disk Space ────────────────────────────────────────────
df -h                              # Human-readable disk usage
df -i                              # Inode usage
df --output=source,fstype,pcent,ipcent,target   # Custom format

# ── I/O Statistics ────────────────────────────────────────
iostat -x 1                        # Extended I/O stats per device
# Key metrics: await (latency), %util (utilization)

# Monitor specific device
iostat -x 1 /dev/nvme0n1

# Real-time I/O by process
iotop                              # Like top, but for I/O
iotop -o                           # Only processes doing I/O

# ── Filesystem Errors ─────────────────────────────────────
dmesg | grep -E "EXT4|XFS|I/O error|blk_update_request"
journalctl -k | grep -i "error\|corrupt\|fail"

# ── Throughput ────────────────────────────────────────────
# Read throughput of a disk
dd if=/dev/nvme0n1 of=/dev/null bs=1M count=1000 status=progress

# Write throughput
dd if=/dev/zero of=/data/test bs=1M count=1000 oflag=direct status=progress
rm /data/test

# ── Filesystem Health ─────────────────────────────────────
smartctl -a /dev/nvme0n1           # SMART data (predictive failure)
smartctl -t short /dev/sda         # Run short SMART test

8.7 Production Scenarios

Scenario 1: Disk Full — Root Cause Analysis

# Alert fires: /data filesystem at 100%

# Step 1: Check which directories are largest
du -sh /data/* | sort -h | tail -20

# Step 2: Find large files
find /data -size +1G -type f -printf '%s %p\n' | sort -n | tail -20

# Step 3: Find files that are deleted but still open (common!)
# A process has a deleted file open, so the disk space isn't freed
lsof | grep deleted | awk '{print $1, $7, $9}'
# If found: restarting that process will release the space

# Step 4: Check for log rotation failures
ls -la /var/log/nginx/
# If log files are huge and not rotated → fix logrotate

# Step 5: Emergency space recovery
# Delete old logs (only if you're sure):
find /data/logs -name "*.log" -mtime +30 -delete

# Step 6: Monitor space continuously
watch -n 5 df -h /data

Scenario 2: I/O Bottleneck Investigation

# Symptom: High iowait (%wa in top)

# Step 1: Confirm I/O is the issue
top
# %wa column > 30% = I/O bottleneck

# Step 2: Identify which device
iostat -x 1
# Look for: await > 10ms (bad for OLTP), %util approaching 100%

# Step 3: Which process is doing the I/O
iotop -o -b -n 3  # 3 samples, non-interactive
# or:
pidstat -d 1 5    # disk I/O per process, 5 samples

# Step 4: What files are being accessed
lsof -p <high_io_pid>

# Step 5: Analyze the I/O pattern
# High read I/O + high await = cache miss (not enough RAM for page cache)
# High write I/O + sequential = normal large write (tune with noatime, etc.)
# High write I/O + random = possible metadata sync issues

# Fix for write-heavy workloads:
# Add noatime,data=writeback to mount options (ext4)
# Or use XFS with delayed allocation

8.8 Interview Questions

Q1: What is the VFS and why does Linux have it?

Answer: The VFS (Virtual File System) is an abstraction layer in the kernel that provides a unified API for different filesystem types. Without VFS, applications would need to know whether they’re reading from an ext4 disk, an NFS share, or a tmpfs. With VFS, all filesystems implement the same set of operations (open, read, write, stat, etc.). Applications call standard syscalls, VFS routes to the appropriate filesystem driver. VFS also maintains key caches: the dentry cache (directory name lookups), the inode cache (file metadata), and the page cache (file data).

Q2: How does OverlayFS work and what is copy-on-write?

Answer: OverlayFS merges multiple filesystem layers into a unified view. Lower layers are read-only (Docker image layers). The upper layer is read-write (container layer). When a process reads a file, OverlayFS searches top-down through layers and returns the first match. When a process writes to a file that exists only in a lower layer, OverlayFS first copies the file from the lower layer to the upper layer (copy-on-write), then modifies it in the upper layer. The lower layers remain unchanged. This enables Docker to share base image layers between multiple containers (each container only needs its own upper layer), saving significant disk space.

Q3: What are inodes and what happens when they run out?

Answer: Inodes are data structures that store file metadata: permissions, ownership, timestamps, size, and pointers to data blocks. Every file has exactly one inode. Directories are files too, containing mappings from filenames to inode numbers (dentries). When you run out of inodes, you cannot create new files even if there’s free disk space, because each new file needs an inode. This commonly happens with mail spools, tmp directories, or applications that create many small files. Fix: delete the small files. Prevention: when creating a filesystem with mkfs.ext4 -i <bytes-per-inode>, use a smaller value (more inodes) for workloads with many small files.

8.9 Summary

Concept	Key Point
VFS	Abstraction layer for all filesystem types
inode	File metadata (not the name)
dentry	Filename → inode mapping
ext4	Default Linux FS, journaled, good for mixed workloads
XFS	High performance, large files, parallel I/O
OverlayFS	Layered FS used by Docker containers
Copy-on-Write	Write copies file from lower layer to upper first
noatime	Don’t update access time — significant I/O savings

Next Chapter: Chapter 9: Device Drivers & Kernel Modules

Prerequisites

Chapter 1 (Linux Architecture) — understand VFS and the file abstraction.

Advantages & Disadvantages

Advantages: VFS allows uniform access to any storage type, robust journaling (ext4) prevents corruption. Disadvantages: High overhead for millions of tiny files, VFS layer adds slight latency.

Common Mistakes

Assuming data is written to disk after write() — data is in page cache until fsync() is called. Power loss without fsync = data loss.
Using ext4 where XFS is better — XFS scales better for large files and high concurrency; ext4 better for many small files.
Treating OverlayFS as a real filesystem — it layers directories; writes go to the upper layer; removes are whiteout files.
Forgetting inode exhaustion is a real problem — df -i to check. Systems with millions of small files exhaust inodes before disk space.
Not setting noatime on busy filesystems — updating access time on every read doubles write I/O on read-heavy workloads.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Filesystem full but df shows space	Inode exhaustion (too many files)	`df -i; ls /proc/sys/fs/inode-nr`	Delete small files in bulk; increase inode count at mkfs time: mkfs.ext4 -i 4096
Can’t delete a file despite being owner	File has immutable attribute set	`lsattr filename`	chattr -i filename; then rm it
fsck needed but filesystem won’t unmount	Root or busy filesystem	`Remount ro: mount -o remount,ro /; or boot to rescue`	Run fsck from rescue media; e2fsck -f /dev/sda1
OverlayFS writes not persisting	Writing to read-only lower layer	`mount	grep overlay; check upperdir`

Hands-on Lab

Objective: Inspect filesystem internals in depth.

# 1. Check disk space and inodes
df -h && df -i

# 2. Inspect filesystem features
tune2fs -l /dev/sda1 | grep -E "features|block|inode"   # ext4
xfs_info /dev/sda2                                         # xfs

# 3. View inode for a file
stat /etc/passwd
# Notice: inode number, hard links, block count

# 4. Create hard link vs symbolic link
ln /etc/hosts /tmp/hosts_hard         # hard link
ln -s /etc/hosts /tmp/hosts_soft      # symlink
stat /tmp/hosts_hard                  # same inode as /etc/hosts
stat /tmp/hosts_soft                  # different inode, points to /etc/hosts

# 5. Inspect OverlayFS (Docker)
docker inspect $(docker ps -q | head -1) | grep -A5 GraphDriver

# 6. strace filesystem calls
strace -e trace=openat,read,write,fsync cat /etc/hostname 2>&1

# 7. Benchmark fsync vs no-fsync
time dd if=/dev/zero of=/tmp/test bs=4k count=10000
time dd if=/dev/zero of=/tmp/test bs=4k count=10000 oflag=sync

Exercises

Find the top 10 directories consuming the most inodes on your system: for d in /var /tmp /home; do echo -n "$d: "; find $d | wc -l; done.
Create a filesystem with a custom inode ratio (mkfs.ext4 -i 1024 /dev/loop0) on a loopback device. Test filling it with tiny files.
Mount an OverlayFS manually: create lowerdir, upperdir, workdir, and mount with mount -t overlay. Create a file and verify it appears in upperdir.
Write a Python script that opens a file, writes data, and calls os.fsync(). Verify with strace that the fsync syscall is issued.
Check for filesystem errors: tune2fs -l /dev/sda1 | grep 'Mount count\|Check interval'. Force a check if needed.

Revision Notes

VFS is the abstraction layer — open(), read(), write() work identically on ext4, XFS, NFS, procfs, sysfs.
Inodes store metadata (permissions, timestamps, size, block pointers) — NOT the filename. Directories map names → inodes.
Page cache: writes go to cache first, flushed to disk by pdflush/writeback. fsync() forces immediate disk write.
OverlayFS layers: lowerdir (read-only base) + upperdir (writes) + workdir (atomic ops) = merged view. Used by Docker.
XFS strengths: parallel I/O, online resize, large files. ext4 strengths: small files, mature, widely supported.
noatime mount option: don’t update access time on reads — significant I/O reduction on read-heavy workloads.

Filesystem Internals: VFS, OverlayFS, ext4, XFS

Chapter 8: Filesystem Internals: VFS, OverlayFS, ext4, XFS

Learning Objectives

Prerequisites

What is this? (Beginner On-Ramp)

8.1 The VFS: Virtual File System

Key VFS Objects

8.2 ext4: The Workhorse Filesystem

ext4 Layout

ext4 Key Features

Inode Exhaustion

8.3 XFS: High-Performance Filesystem

8.4 OverlayFS: How Container Layers Work

OverlayFS Performance

8.5 Filesystem Mount Options for Production

8.6 Filesystem Monitoring

8.7 Production Scenarios

Scenario 1: Disk Full — Root Cause Analysis

Scenario 2: I/O Bottleneck Investigation

8.8 Interview Questions

8.9 Summary

Next Chapter: Chapter 9: Device Drivers & Kernel Modules

Prerequisites

Advantages & Disadvantages

Common Mistakes

Troubleshooting

Hands-on Lab

Exercises

Revision Notes

Further Reading

Related Chapters