Memory Performance

Chapter 64: Memory Performance

Comprehensive Linux Memory Performance Monitoring and Tuning

---

Why This Matters in DevOps/SRE

Memory performance is critical for DevOps and SRE roles because applications rely on proper memory management to function efficiently. Memory leaks, OOM (Out of Memory) kills, and excessive swapping can cause service outages, slow response times, and customer-impacting incidents.

┌─────────────────────────────────────────────────────────────────────────────┐
│                    MEMORY IN DEVOPS/SRE CONTEXT                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    INCIDENT RESPONSE                                │    │
│  │                                                                       │    │
│  │   "Server unresponsive"  ──►  Check memory ──►  OOM killer active   │    │
│  │                                            │                        │    │
│  │                                            ▼                        │    │
│  │                                    Identify memory hog             │    │
│  │                                            │                        │    │
│  │                                            ▼                        │    │
│  │                                    Restart service / scale up      │    │
│  │                                                                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    PROACTIVE MONITORING                             │    │
│  │                                                                       │    │
│  │   Memory Usage > 80%  ──►  Alert on-call  ──►  Scale application    │    │
│  │                                                                       │    │
│  │   Swap usage rising  ──►  Investigate leak  ──►  Schedule fix       │    │
│  │                                                                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    CAPACITY PLANNING                                 │    │
│  │                                                                       │    │
│  │   Current: 16GB/app   │   Project growth: 3x   │   Plan: 64GB node   │    │
│  │                                                                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Real-world scenarios:

• Pod eviction in Kubernetes: When a node runs out of memory, the kubelet evicts pods, causing service disruptions
• Database OOM: MySQL/PostgreSQL getting killed by OOM killer due to large queries
• Java heap issues: JVM consuming excessive RSS beyond heap size
• Redis memory fragmentation: Causing unexpected memory pressure

---

57.1 Linux Memory Architecture

Memory Management Overview

┌────────────────────────────────────────────────────────────────────────┐
│                    LINUX MEMORY ARCHITECTURE                            │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  ┌──────────────────────────────────────────────────────────────────┐ │
│  │                     VIRTUAL MEMORY                                │ │
│  │                                                                   │ │
│  │  ┌──────────────────┐  ┌──────────────────┐                   │ │
│  │  │ Process A        │  │ Process B        │                   │ │
│  │  │                  │  │                  │                   │ │
│  │  │ Virtual Address │  │ Virtual Address │                   │ │
│  │  │ Space 0-4GB    │  │ Space 0-4GB    │                   │ │
│  │  │                  │  │                  │                   │ │
│  │  └────────┬─────────┘  └────────┬─────────┘                   │ │
│  │           │                       │                              │ │
│  │           ▼                       ▼                              │ │
│  │  ┌──────────────────────────────────────────────────────────┐    │ │
│  │  │              PAGE TABLES (MMU)                           │    │ │
│  │  │  Translates virtual addresses to physical addresses     │    │ │
│  │  └──────────────────────────────────────────────────────────┘    │ │
│  │                            │                                      │ │
│  └────────────────────────────┼──────────────────────────────────────┘ │
│                               │                                         │
│  ┌────────────────────────────┼──────────────────────────────────────┐ │
│  │                     PHYSICAL MEMORY                               │ │
│  │                            │                                      │ │
│  │  ┌──────────┬──────────┬───┴───┬──────────┬──────────┐        │ │
│  │  │Page Cache│Buffers  │ Free  │  Slab    │ Process   │        │ │
│  │  │(file data)│(blk dev)│ Pages │(kernel) │   RSS    │        │ │
│  │  └──────────┴──────────┴───────┴──────────┴──────────┘        │ │
│  │                                                                   │ │
│  └───────────────────────────────────────────────────────────────────┘ │
│                               │                                         │
│                               ▼                                         │
│  ┌───────────────────────────────────────────────────────────────────┐ │
│  │                         SWAP                                      │ │
│  │  ┌──────────┬──────────┬──────────┬──────────┐                │ │
│  │  │ Anon     │ Swap     │          │          │                │ │
│  │  │ (process)│ Cache    │          │          │                │ │
│  │  └──────────┴──────────┴──────────┴──────────┘                │ │
│  └───────────────────────────────────────────────────────────────────┘ │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Memory Types in Linux

Type	Description	Location
RSS	Resident Set Size - actual physical memory used	Physical RAM
VSZ	Virtual Size - total virtual memory allocated	Virtual
Page Cache	Cached file content from disk	Physical RAM
Buffers	Block device buffers	Physical RAM
Slab	Kernel data structures	Physical RAM
Swap	Pages moved to disk	Swap space

---

57.2 Memory Information Commands

Viewing Memory

# Basic memory info
free -h
# Output:
#               total        used        free      shared  buff/cache   available
# Mem:           15Gi       4.5Gi       8.2Gi      200Mi       2.3Gi        10Gi
# Swap:         2.0Gi          0B       2.0Gi

# Detailed memory info
cat /proc/meminfo
# MemTotal:       16384000 kB
# MemFree:        8500000 kB
# MemAvailable:   10500000 kB
# Buffers:         240000 kB
# Cached:         2400000 kB
# SwapCached:       10000 kB
# Active:         3200000 kB
# Inactive:       1800000 kB
# SwapTotal:      2097152 kB
# SwapFree:       2097152 kB
# Dirty:            1000 kB
# Writeback:          100 kB

# Memory usage in MB
free -m

# Continuous monitoring
free -h -s 3

# Human-readable with timestamp
free -hwt

Detailed Memory Information

# Hardware memory info
sudo dmidecode -t memory

# Memory devices
sudo dmidecode -t 17

# NUMA memory info
numactl --hardware
numactl --show

# Memory topology
cat /proc/buddyinfo

# VM memory stats
cat /proc/vmstat

# Memory pressure
cat /proc/pressure/memory

---

57.3 Memory Monitoring Tools

top and htop

# Memory usage with top
top
# Press M to sort by memory
# Press m to toggle memory display

# htop with memory
htop
# F6 to sort by %MEM

# Memory info in top
VIRT   - Virtual memory
RES    - Resident (physical) memory
SHR    - Shared memory
%MEM   - Memory percentage

vmstat and sar

# Memory and swap statistics
vmstat 1

# Detailed memory stats
vmstat -s
vmstat -m

# Memory stats with sar
sar -r 1 5
# kbmemfree  kbmemused  %memused  kbbuffers  kbcached  kbcommit   %commit
# 8000000    8000000      50%      200000    2400000  1000000      6%

# Swap activity
sar -S 1 5

# Page statistics
sar -B 1 5

# Slab statistics
sar -m 1 5

Process Memory Analysis

# Memory by process
ps aux --sort=-%mem | head -20

# Detailed process memory
pmap -x 12345
# Shows each memory mapping for process

# Process memory in tree format
ps -eo pid,ppid,%mem,%cpu,comm --forest | head -20

# Memory used by user
ps -U username -o pid,vsz,rss,pmem,comm

# Shared memory segments
ipcs -m

# Shared memory limits
ipcs -l

slabtop

# Kernel slab allocator
slabtop
# Updates every second

# Options
slabtop -s c  # Sort by cache
slabtop -s b  # Sort by object count

---

57.4 Memory Tuning Parameters

sysctl Parameters

# View current values
sysctl vm.swappiness
sysctl vm.vfs_cache_pressure
sysctl vm.overcommit_memory
sysctl vm.overcommit_ratio

# Add to /etc/sysctl.d/99-memory.conf:

# Swappiness (0-100, lower = less swapping)
vm.swappiness = 10

# Cache reclaim pressure (higher = reclaim more cache)
vm.vfs_cache_pressure = 50

# Memory overcommit (0=heuristic, 1=always, 2=never)
vm.overcommit_memory = 0

# Overcommit ratio (when overcommit_memory=2)
vm.overcommit_ratio = 50

# Minimum free memory
vm.min_free_kbytes = 65536

# Dirty page ratios
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
vm.dirty_expire_centisecs = 3000

# Apply changes
sudo sysctl -p

Swappiness Guide

Value	Behavior	Best For
0	Swap only when out of memory	Desktop with SSD, enough RAM
10	Minimal swapping	Most servers
30	Default (some distros)	General use
60	Aggressive	Systems with limited RAM
100	Maximum swapping	Rarely used

Huge Pages

# View huge pages
cat /proc/meminfo | grep Huge

# Set number of huge pages
sudo sysctl vm.nr_hugepages=128

# Make persistent
echo "vm.nr_hugepages = 128" >> /etc/sysctl.d/99-hugepages.conf

# Applications can use huge pages
# PostgreSQL, Oracle: shared_buffers
# Java: -XX:+UseLargePages

Dropping Caches

# Sync first (important!)
sync

# Drop page cache only
echo 1 | sudo tee /proc/sys/vm/drop_caches

# Drop dentries and inodes
echo 2 | sudo tee /proc/sys/vm/drop_caches

# Drop all caches
echo 3 | sudo tee /proc/sys/vm/drop_caches

# Add to cron for regular clearing (not recommended in production)
# 0 3 * * * /bin/sync; /bin/echo 3 > /proc/sys/vm/drop_caches

---

57.5 Swap Management

Creating Swap Space

# Create swap file
sudo dd if=/dev/zero of=/swapfile bs=1M count=2048
# or
sudo fallocate -l 2G /swapfile

# Set permissions
sudo chmod 600 /swapfile

# Format as swap
sudo mkswap /swapfile

# Enable swap
sudo swapon /swapfile

# Verify
swapon -s
free -h

# Add to /etc/fstab for persistence
# /swapfile none swap sw 0 0

Swap Partition

# Create swap partition with fdisk
sudo fdisk /dev/sda
# n (new), p (primary), +2G, t (type), 82 (Linux swap)

# Format
sudo mkswap /dev/sda3

# Enable
sudo swapon /dev/sda3

# Add to /etc/fstab
# /dev/sda3 none swap sw 0 0

Managing Swap

# Disable swap
sudo swapoff /swapfile

# Enable swap
sudo swapon /swapfile

# Priority (higher = used first)
sudo swapon -p 100 /swapfile

# In /etc/fstab:
# /swapfile none swap sw,pri=100 0 0

# Check swap usage
swapon --show
cat /proc/swaps

# Remove swap file
sudo swapoff /swapfile
sudo rm /swapfile

---

57.6 NUMA Management

NUMA Tuning

# Check NUMA configuration
numactl --hardware

# Show current NUMA policy
numactl --show

# Run process on specific node
numactl --cpunodebind=0 --membind=0 process

# Interleave memory across nodes
numactl --interleave=all process

# Move process to node
numactl --membind=1 -p <pid>

Zone Reclaim

# Disable zone reclaim (for NUMA)
sysctl -w vm.zone_reclaim_mode=0

# Make persistent
echo "vm.zone_reclaim_mode = 0" >> /etc/sysctl.d/99-numa.conf

---

57.7 Memory Issues and Troubleshooting

Out of Memory (OOM)

# Check OOM events
dmesg | grep -i "out of memory"
dmesg | grep -i "killed process"

# View OOM history
sudo journalctl -xe | grep -i "killed process"

# Check process killed
dmesg | tail -100 | grep -i oom

# Check vm.overcommit_memory setting
sysctl vm.overcommit_memory

Finding Memory Hogs

# Top memory consumers
ps aux --sort=-%mem | head -10

# Detailed process memory
pmap -x $(pgrep -f processname)

# Memory used by user
ps -eo user,pcpu,pmem,rss,comm --sort=-rss | head

# Shared memory
ipcs -m

# Memory mapped files
lsof +L1 | head

# Check for memory leaks
watch -n 1 'ps -eo pid,vsz,rss,pmem,comm --sort=-rss | head'
# Watch for steadily increasing RSS

Checking Memory Pressure

# Memory pressure (cgroup v2)
cat /proc/pressure/memory

# Some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0

# Using psi (Pressure Stall Information)
cat /proc/pressure/io

# From cgroup
cgget -g memory:/ <cgroup_name>

Troubleshooting Low Memory

# Check what's using memory
free -h
cat /proc/meminfo

# Process memory
ps aux | sort -k4 -r | head

# Kernel memory
slabtop

# Page cache
echo 3 | sudo tee /proc/sys/vm/drop_caches

# Swap usage
swapon -s
vmstat 1

---

57.8 Production Memory Configuration

Database Server

/etc/sysctl.d/99-database.conf

# Memory settings
vm.swappiness = 10
vm.overcommit_memory = 2
vm.overcommit_ratio = 80

# Huge pages
vm.nr_hugepages = 256

# Page cache
vm.vfs_cache_pressure = 50

# Min free
vm.min_free_kbytes = 65536

# Apply
sudo sysctl -p

Web Server

/etc/sysctl.d/99-webserver.conf

vm.swappiness = 60
vm.vfs_cache_pressure = 100
vm.overcommit_memory = 0
vm.dirty_ratio = 60
vm.dirty_background_ratio = 10

High Performance Computing

/etc/sysctl.d/99-hpc.conf

vm.swappiness = 3
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
vm.nr_hugepages = 1024
vm.min_free_kbytes = 1048576
vm.zone_reclaim_mode = 0

---

57.9 Interview Questions

Q1: What is the difference between RSS and VSZ?

Answer:

• RSS (Resident Set Size): Physical memory currently used by the process
• VSZ (Virtual Size): Total virtual memory allocated by the process (may include swapped out pages)

RSS is the actual physical RAM being used. VSZ includes memory that might be on disk (swap).

Q2: What is swappiness and what value should you use?

Answer: vm.swappiness controls how aggressively the kernel swaps pages to disk:

• Range: 0-100
• Higher values = more aggressive swapping
• Lower values = prefer keeping data in RAM

Recommended values:

• 10-30 for servers with ample RAM
• 0 for databases (let them manage memory)
• 60 for systems with limited RAM

Q3: How do you check memory usage in Linux?

Answer:

# Basic
free -h
cat /proc/meminfo

# Per process
ps aux --sort=-%mem
top
htop

# Detailed per process
pmap -x <pid>

# Over time
vmstat 1
sar -r 1

Q4: What is the OOM killer?

Answer: The Out-of-Memory (OOM) killer is a Linux kernel mechanism that kills processes when memory is exhausted. It:

1. Calculates which process to kill (based on oom_score)
2. Kills the selected process to free memory
3. Logs the event to dmesg

The oom_score is influenced by:

• Memory usage
• Process age
• Nice value
• oom_score_adj tunable

Q5: What is memory overcommit?

Answer: Memory overcommit allows processes to allocate more virtual memory than physically available. Modes:

• 0 (Heuristic): Kernel uses heuristics (default)
• 1 (Always): Always allow overcommit (dangerous)
• 2 (Never): Never overcommit (limits = swap + RAM × overcommit_ratio)

Databases typically use mode 2 for predictable behavior.

Q6: How do you create a swap file?

Answer:

# Create 2GB swap file
sudo dd if=/dev/zero of=/swapfile bs=1M count=2048
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Add to /etc/fstab
# /swapfile none swap sw 0 0

Q7: What is page cache?

Answer: Page cache is memory used by the kernel to cache file content from disk. Benefits:

• Reduces disk I/O
• Improves file read performance
• Pages are written back to disk when dirty

Managed by the kernel automatically. Can be controlled with vm.vfs_cache_pressure.

Q8: How do you diagnose a memory leak?

Answer:

1. Monitor memory over time:

   watch -n 1 'ps -eo pid,vsz,rss,pmem,comm --sort=-rss'

2. Compare RSS at different times - steadily increasing indicates leak

3. Check for growing memory in specific process

4. Use tools like:

   • valgrind for application-level analysis
   • pmap to see memory mappings
   • /proc/<pid>/smaps for detailed mapping

---

Quick Reference

Commands

# View memory
free -h
cat /proc/meminfo

# Monitor
vmstat 1
sar -r 1
top
htop

# Process
ps aux --sort=-%mem
pmap -x <pid>

# Swap
swapon -s
swapon/swapoff

# Tune
sysctl -w vm.swappiness=10
echo 3 > /proc/sys/vm/drop_caches

Key Parameters

Parameter	Default	Recommended	Description
vm.swappiness	60	10-30	Swap tendency
vm.overcommit_memory	0	0/2	Overcommit mode
vm.vfs_cache_pressure	100	50	Cache reclaim
vm.dirty_ratio	20	15-40	Dirty page %

---

Common Mistakes & Anti-Patterns

1. Ignoring Memory Pressure Until OOM

❌ WRONG: Only reacting when services start crashing

# Waiting for OOM before taking action
# dmesg shows: "Out of memory: Killed process..."
sudo dmesg | tail -20  # Only checking AFTER crash

✅ CORRECT: Proactive monitoring with alerts

# Set up alert at 80% memory usage
#!/bin/bash
THRESHOLD=80
MEM_USED=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100}')
if [ "$MEM_USED" -gt "$THRESHOLD" ]; then
    curl -X POST "https://hooks.slack.com/services/..." \
        -d "text=":warning: Memory at ${MEM_USED}% on $(hostname)"
fi

2. Setting swappiness to 0

❌ WRONG: Disabling swap completely

# Never do this - can cause OOM更快
sysctl -w vm.swappiness=0

✅ CORRECT: Lower but not zero, depends on workload

# For databases, reduce but don't eliminate
sysctl -w vm.swappiness=10

# Make permanent
echo 'vm.swappiness=10' >> /etc/sysctl.conf

3. Not Understanding RSS vs Cache

❌ WRONG: Treating cache as used memory

# Wrong interpretation - cache is reclaimable
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       28Gi       1.0Gi       200Mi       800Mi       2.0Gi
# This looks like only 1GB free, panic!

✅ CORRECT: Understand available = free + buff/cache

# Correct interpretation
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       28Gi       1.0Gi       200Mi       800Mi       2.0Gi
# available (2GB) is what new processes can get + free (1GB) = 3GB usable

4. Clearing Cache at Wrong Time

❌ WRONG: Clearing cache during production hours

# NEVER do this in production - causes massive I/O spike
echo 3 > /proc/sys/vm/drop_caches

✅ CORRECT: Schedule during maintenance window if needed

# Only in maintenance window, after notifying team
# 2am Sunday morning
0 2 * * 0 sync && echo 3 > /proc/sys/vm/drop_caches

5. Overcommitting Memory Without Limits

❌ WRONG: No memory limits on containers

# docker-compose.yml - no memory limit
services:
  app:
    image: myapp:latest
    # Missing: mem_limit

✅ CORRECT: Set appropriate limits

# docker-compose.yml - with memory limits
services:
  app:
    image: myapp:latest
    mem_limit: 512m
    mem_reservation: 256m
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

---

Summary

In this chapter, you learned:

• ✅ Linux memory architecture
• ✅ Memory information commands
• ✅ Memory monitoring tools (top, vmstat, sar)
• ✅ Memory tuning parameters
• ✅ Swap management
• ✅ NUMA management
• ✅ Troubleshooting memory issues
• ✅ Production configurations
• ✅ Interview questions and answers

---

Next Chapter

Chapter 53: Disk I/O Performance

---

Last Updated: February 2026