Cgroups, Namespaces & Container Primitives

Chapter 5: Cgroups, Namespaces & Container Primitives

Learning Objectives

• Understand how namespaces provide process isolation
• Understand how cgroups provide resource control
• Understand how Docker and Kubernetes use these Linux primitives
• Know how to configure cgroups directly for production resource management
• Understand the security implications of namespaces and cgroups

Prerequisites

• Chapter 3: Process Lifecycle
• Chapter 4: Memory Management
• Basic understanding of Docker (see DevOps Tools Guide)

---

What is this? (Beginner On-Ramp)

Cgroups and Namespaces are the two core features that make containers (like Docker) possible. Namespaces hide things—they make a process think it’s alone on the system. Cgroups limit things—they prevent a process from using too much CPU or RAM.

5.1 The Container Myth: “Containers are Magic”

Containers are not a new technology in the OS. They are a combination of two existing Linux kernel features that existed since the early 2000s:

    What a Container Actually Is
    ─────────────────────────────

    Container = Namespace + Cgroup + Filesystem

    ┌─────────────────────────────────────────────────────────┐
    │                     CONTAINER                            │
    │                                                          │
    │  NAMESPACES (isolation):          CGROUPS (limits):     │
    │  • Own PID space (PID 1!)        • Max CPU: 2 cores     │
    │  • Own network stack             • Max RAM: 512MB       │
    │  • Own hostname                  • Max disk I/O: 100MB/s│
    │  • Own user IDs                  • Max processes: 100   │
    │  • Own filesystem view           • Max open files: 1024 │
    │  • Own IPC objects                                      │
    │                                                          │
    │  FILESYSTEM (copy-on-write):                            │
    │  • OverlayFS layers              (see Chapter 8)        │
    └─────────────────────────────────────────────────────────┘

    Docker/Kubernetes are just tools that make
    configuring these three Linux features easy.

---

5.2 Linux Namespaces: Isolation

A namespace wraps a global system resource in an abstraction that makes processes in the namespace think they have their own isolated instance of that resource.

The 8 Linux Namespace Types

Namespace	Flag	Isolates	Kernel Version
PID	CLONE_NEWPID	Process IDs	3.8
Network	CLONE_NEWNET	Network stack (interfaces, routes, iptables)	3.0
Mount	CLONE_NEWNS	Filesystem mount points	2.4.19
UTS	CLONE_NEWUTS	Hostname and domain name	3.0
IPC	CLONE_NEWIPC	SysV IPC, POSIX message queues	3.0
User	CLONE_NEWUSER	User and group IDs	3.8
Cgroup	CLONE_NEWCGROUP	Cgroup root	4.6
Time	CLONE_NEWTIME	System clocks (for containers)	5.6

PID Namespace: Process Isolation

    PID Namespace in Action
    ──────────────────────

    Host:
    systemd (PID=1)
    ├── sshd (PID=100)
    ├── dockerd (PID=200)
    │   └── containerd (PID=201)
    └── containerd-shim (PID=500)
         └── sh (PID=501)     ← This process is in a new PID namespace

    Inside container (new PID namespace):
    sh (PID=1)                ← Same process, different PID!

    The container process:
    - Thinks it's PID 1 (init of the container)
    - Cannot see or signal host processes
    - Cannot see other containers' processes

    But from the host:
    - We can still see the process as PID=501
    - We can kill it, inspect /proc/501/

# Create a new PID namespace (simple example)
sudo unshare --pid --fork --mount-proc /bin/bash
# Now inside: ps shows only this process and bash
# PID=1 is bash!
ps aux

# Check namespaces of a process
ls -la /proc/<PID>/ns/

# Compare container's namespaces to host's
ls -la /proc/1/ns/               # Host PID 1 (systemd)
docker inspect <container> | grep -i namespace
ls -la /proc/$(docker inspect --format '{{.State.Pid}}' <container>)/ns/

Network Namespace: Network Isolation

    Network Namespace Architecture (Docker veth)
    ─────────────────────────────────────────────

    Host Network Namespace:
    ┌──────────────────────────────────────────────────────┐
    │  eth0 (host NIC)                                     │
    │  docker0 (bridge: 172.17.0.1)                        │
    │  veth0a ──────────────────────────────────────┐      │
    │  veth1a ─────────────────────────────────┐    │      │
    └──────────────────────────────────────────│────│──────┘
                                               │    │
    Container 1 Network Namespace:             │    │
    ┌──────────────────────────────────────────│────┘
    │  eth0 (veth0b, IP: 172.17.0.2)  ─────────┘           │
    │  lo (127.0.0.1)                                       │
    │  own iptables rules                                   │
    └───────────────────────────────────────────────────────┘

    Container 2 Network Namespace:
    ┌───────────────────────────────────────────────────────┐
    │  eth0 (veth1b, IP: 172.17.0.3)                        │
    │  lo (127.0.0.1)                                       │
    └───────────────────────────────────────────────────────┘

# List all network namespaces
ip netns list

# Create a network namespace
ip netns add myns

# Run command in a network namespace
ip netns exec myns ip addr show

# See a container's network namespace
docker inspect --format '{{.State.Pid}}' <container>
nsenter -t <PID> --net -- ip addr show

# Check routes inside a container
nsenter -t <PID> --net -- ip route

Mount Namespace: Filesystem Isolation

    Mount Namespace and OverlayFS
    ──────────────────────────────

    Host filesystem:       Container sees:
    /                      /
    ├── etc/               ├── etc/          (container layer)
    ├── bin/               ├── bin/          (image layer)
    ├── home/              └── proc/         (fresh proc mount)
    └── var/
                           /host-data        NOT visible to container
                           (unless explicitly mounted)

# Enter a container's mount namespace
nsenter -t <container_PID> --mount -- ls /

# See all mounts inside a container
nsenter -t <container_PID> --mount -- cat /proc/mounts

# Create a simple isolated environment (demo)
sudo unshare --mount bash
# Now create a new mount that won't affect host
mount tmpfs /tmp -t tmpfs
mount | grep tmpfs
# After exit, host /tmp is unaffected

---

5.3 Cgroups: Resource Control

Control Groups (cgroups) limit, account for, and isolate resource usage of process groups. Think of cgroups as the kernel’s resource enforcement mechanism.

Cgroups v1 vs v2

    Cgroups v1 (legacy, still common):
    - Multiple hierarchies, one per controller
    - /sys/fs/cgroup/memory/
    - /sys/fs/cgroup/cpu/
    - /sys/fs/cgroup/blkio/
    - Complex, inconsistent API

    Cgroups v2 (modern, Kubernetes uses this):
    - Single unified hierarchy
    - /sys/fs/cgroup/
    - All controllers in one place
    - More consistent delegation model
    - Pressure stall information (PSI)

# Check if system uses cgroups v1 or v2
mount | grep cgroup
# cgroupv2: "cgroup2 on /sys/fs/cgroup type cgroup2"
# cgroupv1: "cgroup on /sys/fs/cgroup/memory type cgroup"

# Or check directly:
ls /sys/fs/cgroup/
# v2: contains files like "memory.current", "cpu.max"
# v1: contains directories like "memory/", "cpu/"

Cgroups v2: Deep Dive

    Cgroup Hierarchy (v2)
    ──────────────────────
    /sys/fs/cgroup/
    ├── system.slice/           ← Systemd services
    │   ├── nginx.service/
    │   │   ├── cgroup.procs    ← PIDs in this cgroup
    │   │   ├── memory.current  ← Current memory usage
    │   │   ├── memory.max      ← Memory limit
    │   │   ├── cpu.weight      ← CPU priority (shares)
    │   │   └── cpu.max         ← CPU quota
    │   └── postgresql.service/
    └── user.slice/             ← User sessions
        └── user-1000.slice/

# Inspect a service's cgroup resources
# Find the cgroup for nginx
cat /proc/$(pidof nginx | awk '{print $1}')/cgroup

# View memory usage
cat /sys/fs/cgroup/system.slice/nginx.service/memory.current
# 52428800 (50MB in bytes)

# View CPU stats
cat /sys/fs/cgroup/system.slice/nginx.service/cpu.stat
# usage_usec 12345678
# user_usec 9876543
# system_usec 2469135

# View pressure stall information
cat /sys/fs/cgroup/system.slice/nginx.service/memory.pressure
# some avg10=0.00 avg60=0.00 avg300=0.00 total=0
# full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Configuring Cgroup Limits via systemd

The recommended way to set cgroup limits in production:

# Method 1: systemd service unit
# In /etc/systemd/system/myapp.service:
[Service]
MemoryMax=2G                    # Hard limit (OOM kill at 2GB)
MemoryHigh=1.5G                 # Soft limit (throttled at 1.5GB)
CPUQuota=200%                   # Max 2 CPUs worth of time
CPUWeight=100                   # Relative CPU priority (default=100)
IOWeight=100                    # Relative I/O priority
TasksMax=1000                   # Max processes/threads

# Method 2: systemctl set-property (live change)
sudo systemctl set-property nginx.service MemoryMax=1G
sudo systemctl set-property nginx.service CPUQuota=100%

# Verify limits are applied
systemctl show nginx.service | grep -E "Memory|CPU|Tasks"

# Method 3: Direct cgroup manipulation (advanced)
echo "2G" > /sys/fs/cgroup/system.slice/nginx.service/memory.max
echo "100000 100000" > /sys/fs/cgroup/system.slice/nginx.service/cpu.max
# Format: quota_usec period_usec (100000/100000 = 100% = 1 CPU)
# 200000/100000 = 200% = 2 CPUs

---

5.4 How Docker Uses Namespaces + Cgroups

# When you run: docker run --memory=512m --cpus=2 nginx

# Docker does approximately:
# 1. Creates namespaces:
unshare --pid --net --mount --uts --ipc bash

# 2. Creates cgroup:
mkdir /sys/fs/cgroup/docker/<container_id>
echo "536870912" > /sys/fs/cgroup/docker/<container_id>/memory.max
echo "200000 100000" > /sys/fs/cgroup/docker/<container_id>/cpu.max

# 3. Moves process to cgroup:
echo $$ > /sys/fs/cgroup/docker/<container_id>/cgroup.procs

# 4. Sets up OverlayFS for filesystem
# (see Chapter 8)

# 5. Executes the container entrypoint in the new namespaces

# Verify with a running container:
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' <container>)
# See namespaces:
ls -la /proc/$CONTAINER_PID/ns/
# See cgroup:
cat /proc/$CONTAINER_PID/cgroup
# See limits:
cat /sys/fs/cgroup/docker/<container_id>/memory.max

---

5.5 Kubernetes and Cgroups

Kubernetes (via the container runtime) manages cgroups for each pod:

# Pod spec with resource requests and limits
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: myapp
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"     # Minimum guaranteed (used for scheduling)
        cpu: "250m"         # 250 millicores = 0.25 CPUs
      limits:
        memory: "512Mi"     # Hard limit (OOM kill if exceeded)
        cpu: "1000m"        # Hard limit (throttled, not killed)

    Kubernetes Cgroup Hierarchy (cgroupv2)
    ──────────────────────────────────────
    /sys/fs/cgroup/
    ├── kubepods.slice/
    │   ├── burstable/           ← Pods with requests < limits
    │   │   └── pod<uid>/
    │   │       ├── myapp/       ← Container cgroup
    │   │       │   ├── memory.max = 512Mi
    │   │       │   └── cpu.max = 100000 100000 (1 CPU)
    │   ├── guaranteed/          ← Pods with requests = limits
    │   └── besteffort/          ← Pods with no requests/limits

# Debug Kubernetes cgroup limits
# Find the container's cgroup
kubectl get pod <pod-name> -o yaml | grep -A5 resources

# On the node:
find /sys/fs/cgroup -name "memory.max" | xargs -I{} sh -c 'echo "{}:"; cat {}'

# Check if a container was OOM killed
kubectl describe pod <pod-name> | grep -A5 "OOMKilled\|Reason"
# OOMKilled: true → pod was killed by OOM, increase memory limit

---

5.6 Security Implications

Namespace Escapes

    Container Security Risks
    ─────────────────────────

    ❌ Privileged containers:
       docker run --privileged nginx
       → Container has ALL capabilities
       → Can mount the host filesystem
       → Can see host processes (no PID namespace benefit)
       → Essentially root on the host

    ❌ Mounting host paths:
       docker run -v /:/host nginx
       → Container has full host filesystem access
       → chroot /host and you're root on the host

    ❌ Host network mode:
       docker run --network=host nginx
       → No network isolation
       → Can bind to any port on the host
       → Can sniff all host traffic

    ✓ Rootless containers:
       Run container runtime as non-root user
       → Even if container escapes, attacker is unprivileged
       See Chapter 35: Container Security

# Check if a container is privileged (security audit)
docker inspect <container> | grep -i privileged
# "Privileged": true → DANGEROUS

# Check host mounts
docker inspect <container> | python3 -c "
import json, sys
data = json.load(sys.stdin)[0]
for mount in data.get('Mounts', []):
    if mount['Source'].startswith('/'):
        print(f\"HOST MOUNT: {mount['Source']} -> {mount['Destination']}\")
"

# Check capabilities
docker inspect <container> | grep -A20 CapAdd

---

5.7 Practical: Build a Container from Scratch

Understanding containers by building one manually:

#!/bin/bash
# minimal-container.sh - Create a minimal container without Docker

set -e

ROOTFS=$(mktemp -d)
echo "Creating rootfs at $ROOTFS"

# Create minimal filesystem
mkdir -p $ROOTFS/{bin,proc,sys,dev,tmp}

# Copy bash and its dependencies
cp /bin/bash $ROOTFS/bin/
cp /bin/ls $ROOTFS/bin/
cp /bin/ps $ROOTFS/bin/
# Copy required shared libraries
for lib in $(ldd /bin/bash | grep -o '/lib[^ ]*'); do
    cp --parents $lib $ROOTFS
done

echo "Creating container..."

# Create the container:
# --pid: new PID namespace
# --mount: new mount namespace
# --uts: new hostname
# --ipc: new IPC namespace
# chroot: new filesystem root
sudo unshare \
    --pid \
    --mount \
    --uts \
    --ipc \
    --fork \
    chroot $ROOTFS /bin/bash -c "
        # Mount essential filesystems
        mount -t proc proc /proc
        mount -t sysfs sysfs /sys
        mount -t tmpfs tmpfs /tmp

        # Set hostname
        hostname mycontainer

        # Start a shell
        exec /bin/bash
    "

echo "Container exited"
rm -rf $ROOTFS

---

5.8 Production Scenarios

Scenario 1: Kubernetes Pod Using Too Much CPU (Throttled)

# Symptom: API latency spikes despite low CPU usage reported by app

# Investigation:
# CPU throttling shows up in cgroup stats, not in %CPU
cat /sys/fs/cgroup/kubepods/burstable/pod<uid>/<container>/cpu.stat
# throttled_usec 5000000    ← Pod was throttled for 5 seconds!
# nr_throttled 1250         ← 1250 throttling events

# Or use metrics (if using cAdvisor/Prometheus):
# container_cpu_cfs_throttled_seconds_total

# Fix: Increase CPU limit in pod spec
# resources:
#   limits:
#     cpu: "2000m"           # Was 500m, increase to 2000m

# Or: Switch to requests-only (no limits) and use LimitRange

Scenario 2: Container Memory Leak Investigation

# Pod keeps getting OOM killed (OOMKilled: true)

# Step 1: Find the container's cgroup
NODE=$(kubectl get pod <pod> -o jsonpath='{.spec.nodeName}')
ssh $NODE "find /sys/fs/cgroup -path '*<pod-uid>*' -name 'memory.current'"

# Step 2: Monitor memory growth
ssh $NODE "watch -n 5 'cat /sys/fs/cgroup/.../memory.current'"

# Step 3: Check memory breakdown
ssh $NODE "cat /sys/fs/cgroup/.../memory.stat"
# anon 524288000    ← Anonymous memory (heap, stack)
# file 104857600    ← File cache
# slab 10485760     ← Kernel slabs

# Step 4: Determine if it's a leak or normal behavior
# If anon grows continuously → likely memory leak
# If file grows → just caching (usually fine)

# Fix approaches:
# 1. Increase memory limit (if memory usage is legitimate)
# 2. Add memory limit to prevent OOM of other pods
# 3. Fix the memory leak in code
# 4. Restart pod periodically as band-aid (CronJob)

---

5.9 Interview Questions

Q1: What are namespaces and cgroups? How do they differ?

Answer: Namespaces provide isolation — they make a process think it has its own instance of a global resource. For example, a PID namespace makes a process think it’s PID 1, a network namespace gives it its own network stack. Cgroups provide resource limits — they enforce how much CPU, RAM, disk I/O, and network bandwidth a group of processes can use. Together they form the basis of containers. Docker/Kubernetes are orchestrators that manage these Linux primitives, along with OverlayFS for layered filesystems.

Q2: A container is using more memory than its limit. What happens?

Answer: The kernel’s OOM killer (operating within the cgroup context) will kill a process in the container. For Kubernetes pods, this shows as OOMKilled: true in kubectl describe pod. The memory limit is enforced by the memory controller in cgroups — when the container’s memory usage reaches memory.max, the OOM killer kills a process in the container. The solution depends on the cause: if the memory usage is legitimate, increase the memory limit. If it’s a memory leak, fix the application code. Always set both requests and limits in Kubernetes pod specs.

Q3: What is a privileged container and why is it dangerous?

Answer: A privileged container (run with --privileged or privileged: true in Kubernetes) receives all Linux capabilities and has its security restrictions largely removed. It can mount the host filesystem, load kernel modules, access all host devices, and manipulate host network interfaces. This effectively gives root access to the host system. A compromised privileged container = compromised host. In production, never run privileged containers. If a specific capability is needed, grant only that capability (e.g., CAP_NET_BIND_SERVICE for binding to port 80).

---

5.10 Summary

Feature	What It Does	Kernel Facility
Process isolation	Each container sees its own PIDs	PID namespace
Network isolation	Each container has its own network	Network namespace
Filesystem isolation	Each container has its own root	Mount namespace + OverlayFS
Hostname isolation	Each container has its own hostname	UTS namespace
CPU limits	Max CPU time per container	Cgroup cpu controller
Memory limits	Max RAM per container	Cgroup memory controller
I/O limits	Max disk throughput	Cgroup blkio/io controller

---

Next Chapter: Chapter 6: Signals, IPC, epoll & File Descriptors

Prerequisites

Chapter 3 (Process Lifecycle), Chapter 4 (Memory Management) — understand processes and memory.

---

Advantages & Disadvantages

Advantages: Lightweight virtualization (no hypervisor needed), extremely fast startup for containers. Disadvantages: Less secure than full VMs (shared kernel), complex to configure manually without Docker/Kubernetes.

---

Common Mistakes

• Thinking containers ARE namespaces — containers use namespaces + cgroups + seccomp + capabilities together; none alone makes a container.
• Forgetting cgroup v1 vs v2 differences — v2 is unified hierarchy; many tools (Docker) still default to v1; check with stat -f /sys/fs/cgroup.
• Setting memory limits without swap limits — a container with --memory=512m can still use unlimited swap unless --memory-swap is also set.
• Assuming PID namespace isolation prevents process visibility — a privileged process can still see all processes via /proc if it mounts the host’s /proc.
• Not setting CPU limits in Kubernetes — pods without resources.limits.cpu can starve other pods on the same node (noisy neighbor).

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Container OOMKilled despite available host RAM	cgroup memory limit hit, not host limit	docker stats CONTAINER; cat /sys/fs/cgroup/memory/docker/CID/memory.usage_in_bytes	Increase container —memory limit or fix memory leak in app
Process can see host processes despite PID namespace	Incorrect /proc mount or privileged container	`ls /proc	wc -l inside container (should be small number)`
cgroup hierarchy not applying limits	Using cgroup v1 path but kernel uses v2	stat -fc %T /sys/fs/cgroup — type=tmpfs=v1, cgroup2fs=v2	Migrate to v2 cgroup paths; update tooling
CPU throttling causing latency spikes	CPU quota exhausted before period ends	cat /sys/fs/cgroup/cpu/docker/CID/cpu.stat — check throttled_time	Increase cpu.quota or cpu.period; or remove CPU limits if headroom allows

---

Hands-on Lab

Objective: Create and inspect cgroups and namespaces manually.

# 1. Inspect cgroup version
stat -fc %T /sys/fs/cgroup
# tmpfs = v1; cgroup2fs = v2

# 2. Create a cgroup manually (v2)
sudo mkdir /sys/fs/cgroup/mytest
echo "+memory +cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo "100M" | sudo tee /sys/fs/cgroup/mytest/memory.max
echo $$ | sudo tee /sys/fs/cgroup/mytest/cgroup.procs

# 3. Run in new namespaces
sudo unshare --pid --fork --mount-proc bash
ps aux   # Should only see this bash + ps

# 4. Inspect namespaces of a process
ls -la /proc/$$/ns/

# 5. View Docker container cgroups
docker run --rm --memory=128m --cpus=0.5 alpine sleep 60 &
CONTAINER=$(docker ps -q | head -1)
cat /sys/fs/cgroup/memory/docker/$CONTAINER/memory.limit_in_bytes 2>/dev/null ||   cat /sys/fs/cgroup/docker/$CONTAINER/memory.max

---

Exercises

1. Create a cgroup that limits a process to 100MB RAM. Run stress-ng --vm 1 --vm-bytes 200M in it. Observe what happens.
2. Use unshare --net to create a new network namespace. Verify it has no network interfaces (except lo). Add a veth pair to connect it.
3. Use nsenter to enter the namespaces of a running Docker container. Verify you see its filesystem and process namespace.
4. Write a shell script that uses cgexec to run a command with a 50% CPU quota. Measure its performance vs unconstrained.
5. Explain why a privileged Docker container (--privileged) effectively has no namespace isolation for security purposes.

---

Revision Notes

• Namespaces isolate what a process can see: PID, NET, MNT, UTS, IPC, USER, Cgroup (8 types in Linux).
• cgroups limit and account for resource USAGE: CPU, memory, I/O, network.
• Containers = namespaces (isolation) + cgroups (limits) + seccomp/capabilities (syscall filtering).
• cgroup v2 is the modern unified hierarchy — one tree, all controllers. Docker defaulted to v1 for years.
• Memory cgroup: memory.max = hard limit (OOMKill at this). memory.high = soft limit (throttle but don’t kill).
• CPU cgroup: cpu.max = QUOTA PERIOD — process gets QUOTA microseconds every PERIOD microseconds.

---

Further Reading

• cgroup v2 docs: https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
• Linux namespaces: man 7 namespaces
• Container Security by Liz Rice — excellent cgroups/namespaces deep dive
• Docker internals: https://docs.docker.com/engine/reference/run/

---

Related Chapters

• Chapter 3 — Process lifecycle and PID management
• Chapter 4 — Memory management and limits
• Chapter 35 — Container security using these primitives

---

Last Updated: July 2026