sysctl: vm.*, kernel.*, fs.*, net.*

Chapter 10: sysctl — Kernel Parameter Tuning

Learning Objectives

• Understand sysctl and how it exposes kernel parameters
• Know the critical production tuning parameters for vm, kernel, fs, and net namespaces
• Know how to safely apply, test, and persist sysctl changes
• Understand the trade-offs of each parameter

Prerequisites

• Chapter 4: Memory Management
• Chapter 1: Linux Architecture

---

What is this? (Beginner On-Ramp)

sysctl is the control panel for the Linux kernel. It allows you to change how the core operating system behaves while it is running, without needing to reboot. You can tweak everything from how aggressively Linux uses RAM to how it handles network connections.

10.1 What is sysctl?

sysctl is the interface to tune kernel parameters at runtime without rebooting. Parameters live in /proc/sys/ and are organized hierarchically.

    sysctl Namespace Map
    ─────────────────────
    /proc/sys/
    ├── vm/          ← Virtual memory, page cache, swapping
    ├── kernel/      ← Core kernel behavior, scheduling
    ├── fs/          ← Filesystem, file descriptors, inotify
    └── net/         ← Network stack (TCP, UDP, IP, etc.)
        ├── core/
        ├── ipv4/
        └── ipv6/

# View all kernel parameters
sysctl -a
sysctl -a 2>/dev/null | grep vm.swappiness

# Read a specific parameter
sysctl vm.swappiness
cat /proc/sys/vm/swappiness   # Same thing

# Set a parameter (immediately, not persistent)
sysctl -w vm.swappiness=10
echo 10 > /proc/sys/vm/swappiness   # Same thing

# Make persistent (survives reboot)
# /etc/sysctl.conf or /etc/sysctl.d/99-custom.conf
echo "vm.swappiness = 10" >> /etc/sysctl.d/99-production.conf
sysctl -p /etc/sysctl.d/99-production.conf   # Apply without reboot

# Apply all sysctl.d files
sysctl --system

---

10.2 Virtual Memory Parameters (vm.*)

vm.swappiness

Controls how aggressively the kernel moves memory pages to swap.

    vm.swappiness Values
    ─────────────────────
    0    = Avoid swapping as much as possible
           (swap only to prevent OOM)
    10   = Low swap use (good for databases and servers)
    60   = Default (balanced)
    100  = Aggressively swap to keep as much page cache as possible

    Rule of thumb:
    - Database servers (PostgreSQL, MySQL): vm.swappiness=10
    - Application servers: vm.swappiness=10-30
    - Desktop: vm.swappiness=60 (default OK)
    - Kubernetes nodes: vm.swappiness=0 (swap disabled recommended)

# Set for database servers
echo "vm.swappiness = 10" >> /etc/sysctl.d/99-production.conf

# Check current swap usage
free -h
vmstat -s | grep -E "swap|used"

vm.dirty_* — Write-Back Tuning

These control when the kernel flushes dirty (modified) pages to disk.

    Dirty Page Lifecycle
    ─────────────────────
    Application writes data → Page becomes "dirty"
          │
          │ Wait for dirty_ratio or dirty_expire_centisecs
          ▼
    Kernel pdflush/kworker writes to disk
          │
          ▼
    Page becomes "clean"

    vm.dirty_background_ratio = 5   (% of RAM)
    Kernel starts background writeback when dirty pages
    exceed 5% of RAM. Application continues without blocking.

    vm.dirty_ratio = 20   (% of RAM)
    Application I/O is blocked until dirty pages drop below 20%.
    This is the hard limit that prevents RAM from filling with dirty pages.

    vm.dirty_expire_centisecs = 3000  (30 seconds)
    Pages older than 30s MUST be written back.

    vm.dirty_writeback_centisecs = 500  (5 seconds)
    Background writeback runs every 5 seconds.

/etc/sysctl.d/99-production.conf

# Production tuning for write-heavy workloads
# Lower dirty ratio = more frequent writes = less data loss risk
# but more I/O pressure
vm.dirty_background_ratio = 2       # Start writeback at 2% (default 10%)
vm.dirty_ratio = 10                 # Block apps at 10% (default 20%)

# Higher dirty ratio = less frequent writes = batched I/O (better throughput)
# Use this for append-only workloads (log servers, analytics)
vm.dirty_background_ratio = 10
vm.dirty_ratio = 30
vm.dirty_expire_centisecs = 6000    # 60 seconds

# Monitor dirty page levels
cat /proc/meminfo | grep Dirty
# Dirty: 12800 kB   ← Pages waiting to be written

# Watch for write storms (dirty spikes to vm.dirty_ratio)
watch -n 1 'cat /proc/meminfo | grep -E "Dirty|Writeback"'

vm.overcommit_memory

    vm.overcommit_memory Values
    ────────────────────────────
    0  (default) = Heuristic overcommit
       - Allow reasonable overcommit
       - Reject obvious overcommits (asking for more than RAM+swap)
       - Most applications assume this

    1  = Always overcommit
       - Never refuse memory requests
       - malloc() always succeeds
       - Higher OOM kill risk
       - Used by: some HPC, trading systems

    2  = Never overcommit (strict)
       - Only commit as much as: RAM * vm.overcommit_ratio/100 + Swap
       - malloc() fails if not enough real memory
       - Safer for critical systems (no surprise OOM)
       - Lower performance (conservative allocation)

# For critical financial/medical systems (know exactly what you have):
vm.overcommit_memory = 2
vm.overcommit_ratio = 80           # Commit up to 80% of RAM + all swap

# For general servers (default is usually fine):
vm.overcommit_memory = 0

# Check current memory commitment
cat /proc/meminfo | grep -E "Committed_AS|CommitLimit"
# CommitLimit:  32768000 kB   ← Maximum that can be committed
# Committed_AS: 12345000 kB   ← Currently committed

vm.min_free_kbytes

# Minimum free memory the kernel maintains (emergency reserve)
# Default: ~1% of RAM, automatically calculated
cat /proc/sys/vm/min_free_kbytes

# For production with large RAM (64GB+), increase to prevent
# allocation failures during memory pressure:
vm.min_free_kbytes = 524288    # 512MB for 64GB RAM server

# Too high = wastes RAM
# Too low = OOM kills before page cache can be reclaimed

vm.vfs_cache_pressure

# Controls tendency to reclaim memory from dentries and inodes (VFS cache)
# Default: 100 (balanced)
# Lower = keep more VFS cache (better for metadata-heavy workloads)
# Higher = reclaim VFS cache more aggressively

# For filesystems with many small files:
vm.vfs_cache_pressure = 50     # Keep VFS cache longer

# For memory-pressured systems with large files:
vm.vfs_cache_pressure = 200    # Reclaim VFS cache faster

---

10.3 Kernel Parameters (kernel.*)

kernel.pid_max

# Maximum PID value. When reached, wraps around.
cat /proc/sys/kernel/pid_max
# Default: 32768 (old), 4194304 (modern)

# Increase for systems with many processes (containers, Java apps)
kernel.pid_max = 4194304

kernel.core_pattern

# Control where core dumps go
# Default: "core" (in current directory, dangerous in production)

# Production: send to systemd-coredump or custom dir
kernel.core_pattern = /tmp/core-%e-%p-%t
# %e = executable name, %p = PID, %t = timestamp

# Or use systemd-coredump (recommended):
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

# Enable core dumps for debugging
ulimit -c unlimited
# In systemd: LimitCORE=infinity

kernel.sysrq

# Magic SysRq key — emergency kernel actions
# 0 = disabled
# 1 = all enabled
# Bitmask:
#   1   = enable all
#   2   = enable control of console logging
#   4   = enable keyboard SAK
#   8   = enable kernel dumps
#   16  = enable sync command
#   32  = enable remount read-only
#   64  = enable kill signal
#   128 = enable reboot

# Production: limit to safe operations
kernel.sysrq = 1               # All enabled (for debugging)
kernel.sysrq = 0               # Disabled (hardened systems)

# Emergency use (when system is frozen):
# Alt+SysRq+f → trigger OOM killer
# Alt+SysRq+s → sync filesystems
# Alt+SysRq+u → remount read-only
# Alt+SysRq+b → reboot immediately
# Mnemonic: "Raising Elephants Is So Ultimately Boring" = r,e,i,s,u,b

# Via file (when physical keyboard not available):
echo f > /proc/sysrq-trigger   # OOM killer
echo s > /proc/sysrq-trigger   # Sync

kernel.nmi_watchdog

# NMI (Non-Maskable Interrupt) watchdog detects CPU hangs
# 0 = disabled, 1 = enabled

# Detect CPU lockups (high overhead on VMs)
kernel.nmi_watchdog = 0        # Disable on VMs for performance
kernel.nmi_watchdog = 1        # Enable on bare metal

# Related: panic on hung task
kernel.hung_task_timeout_secs = 120    # Panic if task hung for 120s
kernel.panic_on_hung_task = 1          # Panic instead of just logging

kernel.randomize_va_space

# ASLR (Address Space Layout Randomization)
# 0 = disabled (NEVER do this in production)
# 1 = randomize stack, libraries, mmap
# 2 = also randomize heap (recommended)

kernel.randomize_va_space = 2   # Default, keep this!

# ASLR makes exploits much harder by randomizing memory addresses
# Disabling ASLR is a serious security risk

---

10.4 Filesystem Parameters (fs.*)

fs.file-max

# Maximum number of file handles system-wide
cat /proc/sys/fs/file-max

# Increase for servers with many processes and connections
fs.file-max = 2097152

# Check current usage
cat /proc/sys/fs/file-nr
# 12345 0 2097152
# open_files  unused_handles  max_files

fs.inotify.max_user_watches

# Maximum number of filesystem watches (inotify)
# Used by: VSCode, Dropbox, systemd-journald, Prometheus exporters

cat /proc/sys/fs/inotify/max_user_watches
# Default: 8192 (too low for many applications!)

# Increase for development machines and monitoring tools
fs.inotify.max_user_watches = 524288

# Symptom of exhaustion:
# "inotify watch limit reached"
# "Failed to watch... Too many open files"

fs.aio-max-nr

# Maximum number of asynchronous I/O requests
# Used by: databases (PostgreSQL, MySQL), high-performance apps
fs.aio-max-nr = 1048576    # 1M AIO requests (default 65536)

# Check current AIO usage
cat /proc/sys/fs/aio-nr

---

10.5 Network Parameters (net.*)

Core Network Parameters

/etc/sysctl.d/99-network.conf

# ─── Socket Buffer Sizes ──────────────────────────────────
# TCP receive/send buffer sizes
net.core.rmem_max = 134217728          # Max receive buffer: 128MB
net.core.wmem_max = 134217728          # Max send buffer: 128MB
net.core.rmem_default = 65536          # Default receive buffer: 64KB
net.core.wmem_default = 65536          # Default send buffer: 64KB

# TCP buffer sizes: min, default, max (in bytes)
net.ipv4.tcp_rmem = 4096 87380 134217728    # Min, default, max receive
net.ipv4.tcp_wmem = 4096 65536 134217728    # Min, default, max send

# ─── Network Queue ────────────────────────────────────────
# Max packets queued on a network interface
net.core.netdev_max_backlog = 250000   # Interface receive queue (default 1000)
net.core.somaxconn = 65535             # Max listen backlog (default 128!)

TCP Tuning

# ─── Connection Management ────────────────────────────────
# Time to wait before reusing TIME_WAIT sockets
net.ipv4.tcp_fin_timeout = 15          # FIN_WAIT2 timeout (default 60s)

# Allow TIME_WAIT socket reuse (for same IP+port pair)
net.ipv4.tcp_tw_reuse = 1              # Enable for client-side connections

# ─── Keep-Alive Settings ──────────────────────────────────
net.ipv4.tcp_keepalive_time = 300      # Idle before sending keepalive (default 7200s!)
net.ipv4.tcp_keepalive_intvl = 30      # Interval between keepalives
net.ipv4.tcp_keepalive_probes = 5      # Probes before declaring dead

# ─── SYN Flood Protection ─────────────────────────────────
net.ipv4.tcp_syncookies = 1            # SYN cookies (ALWAYS enable!)
net.ipv4.tcp_max_syn_backlog = 8192    # Max SYN queue (default 256)

# ─── Port Range ────────────────────────────────────────────
net.ipv4.ip_local_port_range = 1024 65535   # Ephemeral port range

# ─── TCP Fast Open ────────────────────────────────────────
net.ipv4.tcp_fastopen = 3              # Enable for both client and server

Connection Tracking (netfilter/iptables)

# Critical for firewalls and NAT
net.netfilter.nf_conntrack_max = 1000000     # Max tracked connections
net.netfilter.nf_conntrack_tcp_timeout_established = 7200   # TCP connection timeout

# Check current connection count
cat /proc/sys/net/netfilter/nf_conntrack_count

# Alert if approaching maximum:
CURRENT=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
MAX=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
PCT=$((CURRENT * 100 / MAX))
[ $PCT -gt 80 ] && echo "WARNING: conntrack at ${PCT}% of max"

# nf_conntrack full = new connections dropped silently!
# Error: "nf_conntrack: table full, dropping packet"

---

10.6 Complete Production sysctl Configuration

/etc/sysctl.d/99-production.conf

# Production Linux Server Tuning
# Test each setting in staging before production!

# ─── Virtual Memory ───────────────────────────────────────
vm.swappiness = 10
vm.dirty_background_ratio = 5
vm.dirty_ratio = 15
vm.min_free_kbytes = 262144
vm.overcommit_memory = 0
vm.panic_on_oom = 0

# ─── Kernel ───────────────────────────────────────────────
kernel.pid_max = 4194304
kernel.randomize_va_space = 2
kernel.sysrq = 0
kernel.core_pattern = /var/crash/core-%e-%p-%t
kernel.nmi_watchdog = 0

# ─── Filesystem ───────────────────────────────────────────
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
fs.aio-max-nr = 1048576

# ─── Network ──────────────────────────────────────────────
# Core
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.netdev_max_backlog = 250000
net.core.somaxconn = 65535

# TCP
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fastopen = 3

# Security
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Connection tracking
net.netfilter.nf_conntrack_max = 1000000

---

10.7 Safe sysctl Deployment Process

safe-sysctl-apply.sh

#!/bin/bash
# Always test sysctl changes safely!

CONFIG_FILE="$1"
BACKUP_FILE="/tmp/sysctl_backup_$(date +%Y%m%d_%H%M%S).conf"

# Step 1: Backup current values
echo "Backing up current sysctl values..."
sysctl -a > "$BACKUP_FILE" 2>/dev/null
echo "Backup saved to $BACKUP_FILE"

# Step 2: Validate the config file
echo "Validating config..."
sysctl -n --system --dry-run 2>&1 | grep -i error
if [ $? -eq 0 ]; then
    echo "WARNING: Errors in config file"
    exit 1
fi

# Step 3: Apply with --load to fail fast
echo "Applying configuration..."
sysctl -p "$CONFIG_FILE"
EXIT_CODE=$?

if [ $EXIT_CODE -ne 0 ]; then
    echo "ERROR: sysctl apply failed. Rolling back..."
    # Restore each value from backup
    while IFS=' = ' read -r key value; do
        sysctl -w "$key=$value" 2>/dev/null
    done < "$BACKUP_FILE"
    exit 1
fi

echo "Configuration applied successfully!"
echo "To make permanent: copy $CONFIG_FILE to /etc/sysctl.d/"
echo "To rollback: sysctl -p $BACKUP_FILE"

---

10.8 Interview Questions

Q1: What does vm.swappiness do and what value would you use for a PostgreSQL server?

Answer: vm.swappiness controls how aggressively the kernel moves memory pages to swap. A value of 0 means “avoid swapping unless absolutely necessary to prevent OOM.” A value of 100 means “aggressively swap to maximize page cache.” For a PostgreSQL server, you want vm.swappiness=10 (low but not 0). The reason: PostgreSQL manages its own shared_buffers cache. You want PostgreSQL’s working set in RAM, not swapped out — swapping PostgreSQL pages would cause severe latency spikes. Setting it to 0 is also fine for PostgreSQL but value of 10 is the typical recommendation. Never set vm.swappiness=60 (default) for database servers.

Q2: What is the impact of net.core.somaxconn and why might the default be problematic?

Answer: net.core.somaxconn sets the maximum number of connections that can be queued waiting to be accepted by listen(). The default is 128, which is dangerously low for modern web services. When the queue fills (e.g., during a traffic spike), new connections are silently dropped with a connection reset. For production web servers: set to 65535 or higher. But this alone isn’t enough — the application also has a backlog parameter in its listen() call (e.g., in nginx: listen 80 backlog=65535). Both the kernel limit AND the application’s listen backlog must be set to handle high concurrency.

Q3: Explain vm.dirty_ratio and vm.dirty_background_ratio. When would you change them?

Answer: Both control when dirty pages (modified data not yet written to disk) are flushed. dirty_background_ratio (default 10%) triggers background writeback without blocking the application. dirty_ratio (default 20%) causes the application to block all writes until dirty pages are flushed. For a write-heavy log aggregator with SSDs: increase both to allow more buffering and reduce write amplification. For a database requiring durability: decrease both to force more frequent writes and reduce data loss window on crash. For a video streaming/large sequential write workload: increase significantly for better throughput.

---

10.9 Summary

Namespace	Key Parameters	Production Values
vm	swappiness	10 (server), 0 (K8s node)
vm	dirty_ratio	10-15%
vm	min_free_kbytes	256MB for 64GB server
kernel	randomize_va_space	2 (always!)
fs	file-max	2M+
net.core	somaxconn	65535
net.ipv4	tcp_fin_timeout	15s
net.ipv4	tcp_syncookies	1 (always!)

---

Next Chapter: Chapter 11: TCP/IP Stack Tuning & Network Performance

Prerequisites

Chapter 1 (Linux Architecture), Chapter 4 (Memory Management) — understand kernel subsystems.

---

Advantages & Disadvantages

Advantages: Apply changes instantly without rebooting, highly granular control. Disadvantages: Changing the wrong parameter can instantly crash the system or degrade performance.

---

Common Mistakes

• Making sysctl changes without testing in staging — wrong vm.overcommit_memory=2 on a production server causes all large allocations to fail.
• Forgetting sysctl changes are not persistent — sysctl -w changes vanish on reboot; persist in /etc/sysctl.d/*.conf.
• Setting net.ipv4.tcp_tw_reuse=1 without understanding implications — safe for outbound connections, but can cause issues for servers.
• Applying load test tuning values to all servers — a database server needs different tuning from a web proxy.
• Setting vm.swappiness=0 — this means swap only when absolutely necessary, not never; swappiness=1 is safer minimum for servers.

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Application getting ENOMEM with RAM available	vm.overcommit_memory=2 and ratio too low	sysctl vm.overcommit_memory vm.overcommit_ratio	Increase overcommit_ratio or switch to overcommit_memory=1
Network connections failing under load	Ephemeral port range exhausted	sysctl net.ipv4.ip_local_port_range; ss -s shows many TIME_WAIT	Expand port range to 1024-65535; enable tcp_tw_reuse
OOM kills despite available swap	vm.swappiness too low; kernel avoids swap	cat /proc/swaps; sysctl vm.swappiness	Increase swappiness to 10-20; check swap is actually mounted
File descriptor limit errors in app	fs.file-max or per-process ulimit too low	sysctl fs.file-max; ulimit -n	Increase fs.file-max; set LimitNOFILE in systemd unit

---

Hands-on Lab

Objective: Apply and verify sysctl tuning for a production server.

# 1. View all current sysctl values
sysctl -a | wc -l   # hundreds of parameters
sysctl -a | grep vm.swappiness

# 2. Apply a setting temporarily
sudo sysctl -w vm.swappiness=10
sudo sysctl -w net.core.somaxconn=65535

# 3. Make it permanent
echo "vm.swappiness = 10" | sudo tee /etc/sysctl.d/99-production.conf
echo "net.core.somaxconn = 65535" | sudo tee -a /etc/sysctl.d/99-production.conf
sudo sysctl -p /etc/sysctl.d/99-production.conf

# 4. Verify network stack tuning
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
sysctl net.core.rmem_max net.core.wmem_max

# 5. Check file descriptor limits
sysctl fs.file-max
cat /proc/sys/fs/file-nr   # opened, freed, max

# 6. View kernel.* settings
sysctl -a | grep "^kernel\." | head -20

---

Exercises

1. Apply the recommended web server sysctl tuning from Chapter 10. Run ab -n 10000 -c 200 against nginx before and after. Compare results.
2. Configure sysctl for a PostgreSQL database server: set vm.dirty_ratio, vm.dirty_background_ratio, kernel.shmmax, kernel.shmall.
3. Set fs.inotify.max_user_watches=524288 for a system running many file watchers (VS Code, webpack). Verify the change.
4. Use sysctl -a 2>/dev/null | diff - <(sysctl -a 2>/dev/null) as a baseline technique. Apply changes and find the diff.
5. Write an Ansible playbook that applies your production sysctl profile to all servers in the web group.

---

Revision Notes

• sysctl modifies kernel parameters at runtime — /proc/sys/ is the filesystem representation.
• Three namespaces: vm.* (memory), kernel.* (core), net.* (network), fs.* (filesystem limits).
• Persist with /etc/sysctl.d/*.conf — loaded at boot by systemd-sysctl.service.
• Critical network tunings: somaxconn, tcp_tw_reuse, buffer sizes (rmem, wmem), port range.
• Critical VM tunings: swappiness, dirty_ratio, overcommit_memory.
• Always test sysctl changes on one server before fleet-wide rollout — wrong settings can crash services.

---

Further Reading

• Linux sysctl docs: https://www.kernel.org/doc/html/latest/admin-guide/sysctl/
• Production sysctl guide: https://www.brendangregg.com/blog/2015-09-23/linux-perf-analysis-in-60-seconds.html
• man 8 sysctl — complete sysctl interface reference
• PostgreSQL kernel tuning: https://www.postgresql.org/docs/current/kernel-resources.html

---

Related Chapters

• Chapter 11 — TCP/IP stack tuning
• Chapter 12 — Memory subsystem tuning
• Chapter 13 — CPU and storage tuning

---

Last Updated: July 2026