HugePages, THP, NUMA & OOM Killer

Chapter 12: HugePages, THP, NUMA & OOM Killer

Learning Objectives

Understand HugePages and when they improve performance
Understand Transparent Huge Pages and their trade-offs
Understand NUMA and how to avoid NUMA-related performance problems
Deep dive into OOM killer configuration

Prerequisites

What is this? (Beginner On-Ramp)

Memory tuning is adjusting how Linux manages RAM for specific workloads. A database server needs different memory rules than a web server. Tuning involves changing how often Linux flushes data to disk, how it uses ‘swap’ space, and how it allocates large chunks of memory.

12.1 HugePages: Large Memory Pages

Normal memory pages are 4KB. HugePages are 2MB (or 1GB for “gigantic pages”).

Why HugePages Matter

    HugePages vs Normal Pages Performance
    ─────────────────────────────────────

    Database with 32GB working set:

    Normal 4KB pages:
    ┌───────────────────────────────────────────────────────┐
    │  32GB / 4KB = 8,388,608 pages                        │
    │  8M page table entries                               │
    │  TLB can hold ~1,500 entries                         │
    │  TLB hit rate: 1,500/8,000,000 = 0.02%!             │
    │  Every memory access = TLB miss = 4 memory accesses  │
    └───────────────────────────────────────────────────────┘

    With 2MB HugePages:
    ┌───────────────────────────────────────────────────────┐
    │  32GB / 2MB = 16,384 pages                           │
    │  16K page table entries                              │
    │  TLB holds 64+ huge page entries                    │
    │  TLB hit rate: much higher                          │
    │  Memory access: mostly TLB hits (1 cycle vs 4+)     │
    └───────────────────────────────────────────────────────┘

    Result: 10-30% better performance for memory-intensive
    applications (databases, JVMs, in-memory caches)

Configuring HugePages

# ── View Current HugePage Status ──────────────────────────
cat /proc/meminfo | grep -i huge
# HugePages_Total:    2048
# HugePages_Free:     1024
# HugePages_Rsvd:      256
# HugePages_Surp:        0
# Hugepagesize:       2048 kB
# Hugetlb:         4194304 kB

# ── Calculate How Many HugePages You Need ─────────────────
# For PostgreSQL: shared_buffers + a bit more
# PostgreSQL shared_buffers = 8GB → need 8GB/2MB = 4096 huge pages
# Add 10% buffer: 4500 huge pages

# ── Allocate HugePages ────────────────────────────────────
# At runtime (may fail if memory is fragmented!):
echo 4500 > /proc/sys/vm/nr_hugepages

# Persistent (in sysctl):
echo "vm.nr_hugepages = 4500" >> /etc/sysctl.d/99-hugepages.conf
sysctl -p /etc/sysctl.d/99-hugepages.conf

# ── IMPORTANT: Allocate at boot time ──────────────────────
# Add to GRUB kernel parameters for reliable allocation:
# hugepages=4500
# This allocates before memory becomes fragmented

# ── Configure PostgreSQL to Use HugePages ─────────────────
# In postgresql.conf:
huge_pages = on              # Require huge pages
# PostgreSQL will use shared memory with huge pages
# for shared_buffers

# ── Configure Java to Use HugePages ──────────────────────
java -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -jar myapp.jar

# ── Check who is using HugePages ──────────────────────────
for pid in /proc/[0-9]*/; do
    pid_num=$(basename $pid)
    vmhwm=$(grep VmHWM $pid/status 2>/dev/null | awk '{print $2}')
    if [ -n "$vmhwm" ] && [ "$vmhwm" -gt 1000000 ]; then
        comm=$(cat $pid/comm 2>/dev/null)
        echo "PID=$pid_num COMM=$comm VmHWM=${vmhwm}KB"
    fi
done

1GB HugePages (Gigantic Pages)

# For extremely memory-intensive workloads (in-memory databases)
# 1GB pages reduce TLB pressure even further

# Allocate at boot (CANNOT be allocated at runtime)
# Add to kernel parameters:
# hugepagesz=1G hugepages=64   # 64GB of 1GB pages

# Check availability
cat /proc/meminfo | grep Huge
ls /sys/kernel/mm/hugepages/
# hugepages-1048576kB  (1GB)
# hugepages-2048kB     (2MB)

12.2 Transparent HugePages (THP)

Transparent Huge Pages try to automatically use 2MB pages for application memory without requiring application changes.

    THP: The Double-Edged Sword
    ────────────────────────────

    THP Promise:
    - Automatic HugePages for any application
    - No code changes required
    - Reduces TLB misses

    THP Problems in Production:

    1. khugepaged: Background thread collapses 4KB pages into 2MB
       → Periodic CPU spikes when collapsing
       → Unpredictable latency (especially bad for Redis, MongoDB)

    2. Memory Bloat:
       → Process uses 1MB of memory → gets 2MB huge page
       → 50% waste for small allocations

    3. Fragmentation:
       → Huge pages get split back into 4KB pages under pressure
       → Complex memory defragmentation operations

    4. Database Issues:
       → PostgreSQL, Oracle, Redis explicitly recommend disabling THP
       → Causes latency spikes and excessive memory usage

# ── View THP Status ───────────────────────────────────────
cat /sys/kernel/mm/transparent_hugepage/enabled
# [always] madvise never
# [always] = THP for all memory (problematic!)
# madvise  = THP only for madvise(MADV_HUGEPAGE) requests (better)
# never    = THP disabled

# ── Disable THP (Recommended for Databases) ───────────────
# Temporary (until reboot):
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Permanent via systemd:
cat > /etc/systemd/system/disable-thp.service << 'EOF'
[Unit]
Description=Disable Transparent Huge Pages
After=sysinit.target local-fs.target
Before=network.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now disable-thp

# ── For Kubernetes Nodes ──────────────────────────────────
# kubelet requires THP to be disabled
# Add to /etc/rc.local or use the systemd service above

# ── Verify ────────────────────────────────────────────────
cat /sys/kernel/mm/transparent_hugepage/enabled
# always madvise [never]   ← [never] = disabled

12.3 NUMA: Non-Uniform Memory Access

NUMA (Non-Uniform Memory Access) is the memory architecture used in multi-socket servers. Each CPU socket has its own “local” RAM, and accessing remote RAM (from another socket) is slower.

    NUMA Architecture (2-socket server)
    ────────────────────────────────────

    ┌─────────────────────────────────────────────────────────┐
    │                        SERVER                            │
    │                                                          │
    │  ┌──────────────────────┐  ┌──────────────────────────┐ │
    │  │     NUMA Node 0      │  │      NUMA Node 1         │ │
    │  │                      │  │                          │ │
    │  │  CPU 0  CPU 1  CPU 2 │  │  CPU 8   CPU 9  CPU 10  │ │
    │  │  CPU 3  CPU 4  CPU 5 │  │  CPU 11  CPU 12 CPU 13  │ │
    │  │  CPU 6  CPU 7        │  │  CPU 14  CPU 15          │ │
    │  │                      │  │                          │ │
    │  │  Local RAM: 64GB     │  │  Local RAM: 64GB         │ │
    │  │  Latency: 50ns       │  │  Latency: 50ns (local)  │ │
    │  └──────────────────────┘  └──────────────────────────┘ │
    │           │                              │               │
    │           └──────────── QPI/UPI ─────────┘               │
    │              Cross-NUMA latency: ~100ns (2x slower!)     │
    └─────────────────────────────────────────────────────────┘

    The NUMA problem:
    If your app runs on CPU 0 (NUMA Node 0) but its memory
    was allocated on NUMA Node 1, every memory access crosses
    the QPI bus at 2x the latency.

# ── View NUMA Topology ────────────────────────────────────
numactl --hardware
# available: 2 nodes (0-1)
# node 0 cpus: 0 1 2 3 4 5 6 7
# node 0 size: 65536 MB
# node 0 free: 32768 MB
# node 1 cpus: 8 9 10 11 12 13 14 15
# node 1 size: 65536 MB
# node 1 free: 30000 MB
# node distances:
# node   0   1
#   0:  10  21   ← Local access=10, cross-NUMA access=21
#   1:  21  10

# View NUMA statistics
numastat
numastat -p <PID>   # NUMA stats for a process

# ── Bind Process to NUMA Node ─────────────────────────────
# Run nginx on NUMA node 0 only
numactl --cpunodebind=0 --membind=0 nginx

# PostgreSQL on NUMA node 0 with local memory
numactl --cpunodebind=0 --localalloc postgres

# ── Systemd NUMA Binding ──────────────────────────────────
# In systemd service file:
NUMAPolicy=bind
NUMAMask=0       # Bind to NUMA node 0

# ── NUMA Policy Options ───────────────────────────────────
# --localalloc: Allocate on the local NUMA node (default for most)
# --membind=0:  Only use memory from NUMA node 0
# --interleave=all: Round-robin memory allocation across nodes
#              (best for applications that don't have NUMA affinity)

# ── Detect NUMA Imbalance ─────────────────────────────────
# High numa_miss = process accesses memory on wrong NUMA node
numastat -p $(pidof postgres) 2>/dev/null
# numa_hit: 12345678    ← Local node accesses
# numa_miss: 5678901    ← Remote node accesses (BAD if high!)

# NUMA miss rate:
numastat | awk '/numa_miss/ {miss=$2} /numa_hit/ {hit=$2} END {
    print "NUMA miss rate:", miss/(hit+miss)*100, "%"
}'

NUMA Interleaving for JVMs

# Java applications often benefit from memory interleaving
# (spreads memory across all NUMA nodes, reducing imbalance)
numactl --interleave=all java -Xmx16g -jar myapp.jar

# Or in systemd:
NUMAPolicy=interleave
NUMAMask=0-1     # All NUMA nodes

12.4 OOM Killer Deep Dive

How the OOM Killer Selects a Victim

    OOM Killer Selection Algorithm
    ────────────────────────────────

    For each process, calculate oom_score:

    oom_score = (pages_used / total_pages) × 1000
                + oom_score_adj              ← Tunable (-1000 to 1000)

    Additional adjustments:
    - Processes with CAP_SYS_PTRACE: score × 0.5 (lower priority)
    - Processes using swap: score × (1 + swap_used/total_swap)

    Kill the process with HIGHEST oom_score

    Example scoring:
    ─────────────────────
    postgres (4GB/64GB RAM, adj=-900): score = (62) + (-900) = -838
    java     (8GB/64GB RAM, adj=0):   score = (125) + 0 = 125
    redis    (2GB/64GB RAM, adj=100): score = (31) + 100 = 131

    Redis would be killed first! (highest score = 131)
    Adjust redis to adj=900 to make it even less preferred:
    redis oom_score = 31 + 900 = 931 → will be killed before others!

# ── View OOM Scores ───────────────────────────────────────
# For a specific process
cat /proc/$(pidof postgres | awk '{print $1}')/oom_score
cat /proc/$(pidof postgres | awk '{print $1}')/oom_score_adj

# For all processes
for pid in /proc/[0-9]*/; do
    pid_num=$(basename $pid)
    score=$(cat $pid/oom_score 2>/dev/null)
    adj=$(cat $pid/oom_score_adj 2>/dev/null)
    comm=$(cat $pid/comm 2>/dev/null)
    [ -n "$score" ] && echo "$score $adj $pid_num $comm"
done | sort -n | tail -20

# ── Set OOM Priority ──────────────────────────────────────
# Protect critical services (systemd or manual):
echo -900 > /proc/$(pidof sshd)/oom_score_adj     # Protect SSH
echo -900 > /proc/$(pidof postgres | awk '{print $1}')/oom_score_adj

# Make cache processes more killable:
echo 500 > /proc/$(pidof redis-server)/oom_score_adj

# ── Systemd Integration ───────────────────────────────────
# In service unit file:
[Service]
OOMScoreAdjust=-900   # -1000 = never kill, +1000 = always kill first

# ── OOM Kill Detection and Response ───────────────────────
# Detect OOM kills
journalctl -k | grep -i "out of memory\|oom_kill_process\|killed process"
dmesg -T | grep -i "oom\|killed process"

# Set up OOM notification (via eBPF or script):
# Simple monitoring script:
while true; do
    if journalctl -k --since "1 minute ago" | grep -q "Out of memory"; then
        echo "OOM KILL DETECTED at $(date)" | mail -s "ALERT: OOM Kill" ops@example.com
        journalctl -k --since "5 minutes ago" | grep -i "oom\|killed" >> /var/log/oom_events.log
    fi
    sleep 60
done &

# ── OOM Prevention ────────────────────────────────────────
# Monitor memory available and alert before OOM
awk '/MemAvailable/ {avail=$2} /MemTotal/ {total=$2} END {
    pct = avail/total*100
    if (pct < 10) {
        print "CRITICAL: Memory available only " pct "%" > "/dev/stderr"
        exit 1
    }
}' /proc/meminfo

Kernel OOM Panic

# For high-availability systems where OOM kill is unacceptable:
# Panic the entire system instead of killing a process
# (ensures cluster can fail over cleanly)

# vm.panic_on_oom = 0  → kill a process (default)
# vm.panic_on_oom = 1  → panic if OOM in user context
# vm.panic_on_oom = 2  → always panic

# Combined with kernel.panic:
kernel.panic = 30        # Reboot 30 seconds after panic
vm.panic_on_oom = 1      # Trigger panic on OOM

# This ensures the cluster detects the failure and fails over
# rather than having a degraded system with a killed critical process

12.5 Memory Bandwidth Monitoring

# Check memory bandwidth usage (requires Intel PCM or perf)
perf stat -e cache-misses,cache-references,LLC-load-misses sleep 5

# Monitor bandwidth with numastat
numastat -m

# Check NUMA statistics via perf
perf stat -a -e node-stores,node-loads sleep 5

# High node-stores with high miss rate = NUMA problem

12.6 Production Scenarios

Scenario 1: Redis Latency Spikes from THP

# Redis latency spikes occurring every ~60 seconds
# Symptom: redis-cli --latency shows periodic 10ms+ spikes

# Investigation:
dmesg | grep -i "huge\|khugepaged"
# khugepaged: huge page allocated
# ← khugepaged is running and disrupting Redis

# Fix:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

# Verify fix:
redis-cli --latency -h localhost -p 6379
# Latency should now be consistently < 1ms

Scenario 2: Dual-Socket Server with NUMA Imbalance

# PostgreSQL running slowly on dual-socket server
# High CPU but queries slower than expected

# Check NUMA topology
numactl --hardware

# Check PostgreSQL's NUMA statistics
numastat -p $(pgrep -f "postgres: bgworker" | head -1)
# numa_miss: 45678901  ← Very high! Cross-NUMA memory accesses

# Solution: Bind PostgreSQL to a single NUMA node
# In postgresql.conf or startup script:
numactl --cpunodebind=0 --membind=0 /usr/lib/postgresql/15/bin/postgres -D /var/lib/postgresql/15/main

# Or in systemd service override:
# ExecStart=/usr/bin/numactl --cpunodebind=0 --membind=0 /usr/lib/postgresql/15/bin/postgres -D /var/lib/postgresql/15/main

# Result: numa_miss drops to near 0, query latency improves 20-40%

12.7 Interview Questions

Q1: What is the difference between HugePages and Transparent Huge Pages?

Answer: HugePages are manually configured 2MB or 1GB memory pages, pre-allocated by the kernel and used only by applications that explicitly request them (via mmap with MAP_HUGETLB, or configured database shared memory). They’re stable, predictable, and never get defragmented. Transparent Huge Pages (THP) are automatic — the kernel tries to use 2MB pages for any application’s memory without requiring explicit requests. THP sounds like a free lunch but causes problems in practice: the background khugepaged thread creates CPU spikes during page collapsing, THP wastes memory for small allocations, and defragmenting huge pages under memory pressure adds latency. Databases (PostgreSQL, Redis, Oracle) explicitly recommend disabling THP.

Q2: What is NUMA and how does it affect database performance?

Answer: NUMA (Non-Uniform Memory Access) means that in multi-socket servers, each CPU socket has its own local RAM, and accessing memory attached to a different socket (remote/cross-NUMA) takes 1.5-3x longer. For a database like PostgreSQL accessing its shared_buffers, if the process runs on one CPU but its memory pages are on another NUMA node, every memory access crosses the QPI/UPI interconnect. This can cause 20-50% performance degradation. Fix: bind the database process to a single NUMA node with numactl --cpunodebind=0 --membind=0. Monitor with numastat -p <PID> — high numa_miss count indicates cross-NUMA access.

12.8 Summary

Feature	What It Does	Production Recommendation
HugePages	2MB pages for less TLB pressure	Enable for databases, JVMs
THP	Auto huge pages	Disable for databases/Redis
NUMA	Multi-socket memory locality	Bind apps to single NUMA node
OOM Killer	Kills processes when RAM full	Set `OOMScoreAdjust=-900` for critical

Next Chapter: Chapter 13: CPU Affinity, IRQ Balancing & Storage Tuning

Prerequisites

Chapter 4 (Memory Management), Chapter 10 (sysctl) — understand virtual memory and sysctl framework.

Advantages & Disadvantages

Advantages: Optimizes database performance (HugePages), controls swap behavior to prevent thrashing. Disadvantages: Hardcoding HugePages locks up RAM, incorrect swap settings can freeze the server.

Common Mistakes

Setting vm.nr_hugepages at runtime on a loaded server — huge page allocation requires contiguous memory; allocate at boot before fragmentation.
Using THP always mode for databases — PostgreSQL and MySQL both recommend never or madvise due to compaction latency spikes.
Ignoring NUMA topology — allocating memory from a remote NUMA node adds 2-3× latency; always check with numastat.
Setting vm.dirty_ratio=80 for database servers — this allows 80% of RAM to become dirty before writeback, causing massive I/O spikes.
Not tuning vm.min_free_kbytes — if set too low, allocations block waiting for memory reclaim; set to ~1% of RAM for servers.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
PostgreSQL latency spikes every few minutes	THP compaction pauses	`grep AnonHugePages /proc/meminfo; dmesg	grep -i compact`
OOM kill despite available swap	Low vm.swappiness causing kernel to prefer OOM kill over swap	`cat /proc/swaps; sysctl vm.swappiness`	Set vm.swappiness=10 (not 0); verify swap is mounted and active
NUMA remote access causing slowness	Process pinned to node 0 but allocating from node 1	`numastat -p PID; numactl --hardware`	Use numactl —localalloc to pin process and memory together
High dirty page accumulation, I/O spike	vm.dirty_ratio too high — pages accumulate then flush all at once	`cat /proc/meminfo	grep Dirty; iostat -x 1`

Hands-on Lab

Objective: Configure and verify memory tuning settings.

# 1. Check NUMA topology
numactl --hardware
numastat

# 2. THP status and configuration
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
# Disable THP for databases:
echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

# 3. Configure Huge Pages
grep Huge /proc/meminfo
sudo sysctl -w vm.nr_hugepages=512
grep Huge /proc/meminfo   # Verify allocation succeeded

# 4. Memory dirty ratios
sysctl vm.dirty_ratio vm.dirty_background_ratio vm.dirty_expire_centisecs

# 5. NUMA-aware process execution
numactl --cpunodebind=0 --membind=0 stress-ng --cpu 4 --timeout 10s

# 6. OOM tuning — protect critical process
echo -1000 | sudo tee /proc/$(pgrep postgres | head -1)/oom_score_adj

Exercises

Configure huge pages for PostgreSQL: set vm.nr_hugepages=256, then configure PostgreSQL huge_pages=on. Verify huge page usage with grep Huge /proc/meminfo.
Compare PostgreSQL query latency with THP always vs never. Run pgbench -i -s 100 && pgbench -c 10 -T 60 both ways.
Use numactl --interleave=all vs --membind=0 for a memory-intensive workload. Measure with numastat after each run.
Set vm.dirty_ratio=5 and vm.dirty_background_ratio=2. Run a write benchmark and compare I/O patterns with default values.
Write a systemd service drop-in that sets MemoryMax=2G and MemorySwapMax=0 for an application. Verify cgroup limits.

Revision Notes

NUMA: memory access from remote socket adds latency. numactl --localalloc ensures allocation from local node.
THP: automatic 2MB pages. Databases need madvise or never — compaction stalls cause query latency spikes.
Dirty page tuning: dirty_background_ratio = start background writeback at X%. dirty_ratio = block processes at Y%.
Huge pages: pre-allocated at boot (vm.nr_hugepages). Applications must use MAP_HUGETLB or SHM_HUGETLB.
vm.min_free_kbytes: kernel always reserves this much free RAM for atomic allocations. Set to ~1% of RAM.
OOM score: 0-1000. Set oom_score_adj=-1000 to protect critical processes from OOM killer.

HugePages, THP, NUMA & OOM Killer

Chapter 12: HugePages, THP, NUMA & OOM Killer

Learning Objectives

Prerequisites

What is this? (Beginner On-Ramp)

12.1 HugePages: Large Memory Pages

Why HugePages Matter

Configuring HugePages

1GB HugePages (Gigantic Pages)

12.2 Transparent HugePages (THP)

12.3 NUMA: Non-Uniform Memory Access

NUMA Interleaving for JVMs

12.4 OOM Killer Deep Dive

How the OOM Killer Selects a Victim

Kernel OOM Panic

12.5 Memory Bandwidth Monitoring

12.6 Production Scenarios

Scenario 1: Redis Latency Spikes from THP

Scenario 2: Dual-Socket Server with NUMA Imbalance

12.7 Interview Questions

12.8 Summary

Next Chapter: Chapter 13: CPU Affinity, IRQ Balancing & Storage Tuning

Prerequisites

Advantages & Disadvantages

Common Mistakes

Troubleshooting

Hands-on Lab

Exercises

Revision Notes

Further Reading

Related Chapters