Advanced Storage: LVM, ZFS, Ceph & NVMe

Chapter 50: Advanced Storage: LVM, ZFS, Ceph & NVMe

Learning Objectives

Manage flexible storage with LVM thin provisioning and snapshots
Use ZFS for data integrity, compression, and native snapshots
Understand Ceph distributed storage architecture and CRUSH maps
Optimize NVMe SSDs for production database workloads

What is this? (Beginner On-Ramp)

Advanced storage technologies (like LVM, ZFS, and Ceph) go far beyond standard hard drives. They allow you to combine dozens of disks into massive pools, take instant snapshots of terabytes of data, and automatically heal corrupted files using checksums.

50.1 LVM: Logical Volume Manager

    LVM Architecture
    ─────────────────

    Physical Disks
    /dev/sda  /dev/sdb  /dev/sdc  /dev/sdd
         │         │        │        │
         ▼         ▼        ▼        ▼
    ┌─────────────────────────────────────────┐
    │  Physical Volumes (PVs)                  │
    │  /dev/sda1  /dev/sdb1  /dev/sdc  /dev/sdd│
    └─────────────────────────────────────────┘
                        │ combined into
                        ▼
    ┌─────────────────────────────────────────┐
    │  Volume Group (VG): "data_vg"           │
    │  Total: 4TB                              │
    └─────────────────────────────────────────┘
                        │ carved into
                        ▼
    ┌─────────┐  ┌──────────┐  ┌─────────────┐
    │  LV:    │  │  LV:     │  │  LV:        │
    │  root   │  │  data    │  │  logs       │
    │  100GB  │  │  2TB     │  │  500GB      │
    └─────────┘  └──────────┘  └─────────────┘

    Key Benefits:
    • Online resize (grow LVs without downtime)
    • Snapshots (for backup, testing)
    • Striping across multiple disks (RAID-like performance)
    • Thin provisioning (allocate on demand)

# ── LVM Setup ─────────────────────────────────────────────
# Initialize physical volumes
pvcreate /dev/sdb /dev/sdc /dev/sdd

# Create volume group
vgcreate data_vg /dev/sdb /dev/sdc /dev/sdd

# Create logical volume
lvcreate -n data_lv -L 1T data_vg        # 1TB standard LV
lvcreate -n logs_lv -L 500G data_vg

# Format and mount
mkfs.xfs /dev/data_vg/data_lv
mount /dev/data_vg/data_lv /data

# ── Online Resize (LIVE, no downtime!) ────────────────────
# Extend LV and filesystem
lvextend -L +500G /dev/data_vg/data_lv   # Add 500GB
xfs_growfs /data                          # Grow XFS online
# OR: resize2fs /dev/data_vg/data_lv      # For ext4

# Add new disk to existing VG
pvcreate /dev/sde
vgextend data_vg /dev/sde
# VG now has extra space

# ── LVM Snapshots ─────────────────────────────────────────
# Create snapshot (for backup)
lvcreate -L 50G -s -n data_snap /dev/data_vg/data_lv

# Mount snapshot (read-only view of data at snapshot time)
mount -o ro /dev/data_vg/data_snap /mnt/snapshot

# Backup from snapshot (original LV still available)
rsync -av /mnt/snapshot/ /backup/

# Remove snapshot
umount /mnt/snapshot
lvremove /dev/data_vg/data_snap

# ── Thin Provisioning ─────────────────────────────────────
# Create thin pool
lvcreate -L 2T --thinpool thin_pool data_vg

# Create thin LVs (over-provision)
lvcreate --thin -n vm01 -V 500G data_vg/thin_pool   # "500GB" LV
lvcreate --thin -n vm02 -V 500G data_vg/thin_pool   # Another "500GB" LV
# Total "provisioned": 1TB, actual pool: 2TB, but LVs use space on demand

50.2 ZFS: The Data Integrity Filesystem

    ZFS Features at a Glance
    ──────────────────────────

    • Copy-on-Write: writes never overwrite existing data
    • End-to-end checksums: every block checksummed (data + metadata)
    • Automatic self-healing: detects and repairs corruption
    • Native compression: LZ4, ZSTD, gzip
    • Native snapshots: O(1), instant
    • Native replication: zfs send | zfs receive
    • ARC cache: adaptive replacement cache (uses all RAM efficiently)
    • Scrubs: offline integrity verification
    • RAIDZ: software RAID with better resilience than RAID5

# ── ZFS Pool Creation ─────────────────────────────────────
# Striped pool (no redundancy)
zpool create data_pool /dev/sdb /dev/sdc

# Mirror (RAID-1)
zpool create data_pool mirror /dev/sdb /dev/sdc

# RAIDZ (like RAID-5, 1 disk fault tolerance)
zpool create data_pool raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde

# RAIDZ2 (2 disk fault tolerance, like RAID-6)
zpool create data_pool raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

# Pool status
zpool status -v data_pool

# ── ZFS Dataset Management ────────────────────────────────
# Create datasets (like directories with properties)
zfs create data_pool/databases
zfs create data_pool/databases/postgres
zfs create data_pool/backups

# List datasets
zfs list

# ── Properties: Compression + ARC ────────────────────────
# Enable LZ4 compression (usually saves 30-50%, near-zero CPU overhead)
zfs set compression=lz4 data_pool/databases

# Enable ZSTD for backup data (better ratio, more CPU)
zfs set compression=zstd data_pool/backups

# Disable access time (reduces writes)
zfs set atime=off data_pool/databases/postgres

# Record size for databases (match PostgreSQL 8K page size)
zfs set recordsize=8K data_pool/databases/postgres

# Check compression ratio achieved
zfs get compressratio data_pool

# ── ZFS Snapshots ─────────────────────────────────────────
# Create snapshot (instant, O(1))
zfs snapshot data_pool/databases/postgres@before_migration

# List snapshots
zfs list -t snapshot

# Rollback to snapshot
zfs rollback data_pool/databases/postgres@before_migration

# Clone snapshot to new dataset
zfs clone data_pool/databases/postgres@before_migration data_pool/databases/postgres_test

# Destroy snapshot
zfs destroy data_pool/databases/postgres@before_migration

# ── ZFS Replication (backup / DR) ────────────────────────
# Send snapshot to remote server
zfs send data_pool/databases/postgres@daily-2024-01-15 | \
    ssh backup-server "zfs receive backup_pool/postgres"

# Incremental send (only changes since last snapshot)
zfs send -i data_pool/databases/postgres@daily-2024-01-14 \
            data_pool/databases/postgres@daily-2024-01-15 | \
    ssh backup-server "zfs receive backup_pool/postgres"

# ── Scrub (integrity check) ───────────────────────────────
zpool scrub data_pool      # Start scrub
zpool status data_pool     # Check progress

# Schedule weekly scrubs (cron)
# 0 2 * * 7 /sbin/zpool scrub data_pool

50.3 Ceph: Distributed Storage

    Ceph Architecture
    ──────────────────

    ┌──────────────────────────────────────────────────────────┐
    │  Client Interfaces                                        │
    │  librados  │  RBD (block)  │  CephFS  │  RGW (S3/Swift) │
    └────────────────────────────┬─────────────────────────────┘
                                 │
    ┌────────────────────────────▼─────────────────────────────┐
    │  RADOS (Reliable Autonomic Distributed Object Store)     │
    │                                                          │
    │  OSD (Object Storage Daemons — one per disk):           │
    │  osd.0  osd.1  osd.2  osd.3  osd.4  osd.5  ...        │
    │  (each OSD manages one disk, stores data objects)       │
    │                                                          │
    │  Monitor Cluster (3+ for quorum):                       │
    │  mon.a  mon.b  mon.c                                     │
    │  (cluster state, CRUSH map, OSD map)                    │
    │                                                          │
    │  Manager (mgr): metrics, dashboard                       │
    │  MDS: metadata server (for CephFS only)                 │
    └──────────────────────────────────────────────────────────┘

    Data Placement (CRUSH Algorithm):
    ────────────────────────────────────
    object → hash → PG (Placement Group) → CRUSH → OSD list
    CRUSH knows: rack topology, failure domains
    → Never places replicas in same rack
    → Pure math, no lookup table, scales to thousands of OSDs

# ── Ceph Health ───────────────────────────────────────────
ceph status              # Overall cluster health
ceph health detail       # Detailed warnings/errors
ceph osd stat            # OSD count and status
ceph df                  # Space usage

# ── Pool Management ───────────────────────────────────────
# List pools
ceph osd lspools

# Create pool (replicated, 3 copies)
ceph osd pool create mypool 128 128 replicated
ceph osd pool set mypool size 3          # 3 replicas
ceph osd pool set mypool min_size 2      # Allow write with 2 copies

# Create pool (erasure coded — better storage efficiency)
ceph osd erasure-code-profile set myprofile k=4 m=2  # 4 data + 2 parity
ceph osd pool create ec_pool 128 128 erasure myprofile

# ── RBD (RADOS Block Device — Kubernetes PVC backend) ─────
# Create RBD image
rbd create --size 100G mypool/myimage

# Map to block device
rbd map mypool/myimage
# → /dev/rbd0

mkfs.xfs /dev/rbd0
mount /dev/rbd0 /mnt/ceph-block

# ── Ceph Cluster Monitoring ───────────────────────────────
# Watch OSD performance
ceph osd perf

# Slow OSD detection
ceph osd blocked-by

# PG distribution (should be even)
ceph pg dump | awk '{print $1, $15}' | sort | head -20

# Rebalancing status
ceph pg stat
# x pgs: y active+clean, z active+remapped  ← rebalancing

50.4 NVMe Optimization

# ── NVMe Identification ───────────────────────────────────
nvme list
nvme id-ctrl /dev/nvme0      # Controller capabilities
nvme smart-log /dev/nvme0    # Health and wear data

# Check for critical warnings
nvme smart-log /dev/nvme0 | grep -i "critical\|warning\|spare\|temp"

# ── NVMe Queue Depth ──────────────────────────────────────
# NVMe supports 64K queues × 64K depth each (vs SATA: 1 queue × 32 depth)
cat /sys/block/nvme0n1/queue/nr_requests    # Current queue depth

# ── I/O Scheduler ────────────────────────────────────────
# NVMe: use none (no scheduler) or mq-deadline
cat /sys/block/nvme0n1/queue/scheduler

# Set none scheduler for NVMe (it handles its own scheduling)
echo none > /sys/block/nvme0n1/queue/scheduler
# OR mq-deadline (for mixed read/write workloads):
echo mq-deadline > /sys/block/nvme0n1/queue/scheduler

# Make persistent via udev rule:
# /etc/udev/rules.d/60-nvme-scheduler.rules:
# ACTION=="add|change", KERNEL=="nvme*", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="none"

# ── Write Caching and Barriers ────────────────────────────
# Check write cache status
hdparm -I /dev/nvme0n1 | grep "Write cache"

# For database workloads: disable OS write cache (let NVMe handle)
# PostgreSQL: use O_DIRECT, fsync for durability

# ── NVMe Namespaces (Enterprise NVMe) ────────────────────
# Enterprise NVMe supports multiple namespaces (logical divisions)
nvme list-ns /dev/nvme0      # List namespaces
nvme id-ns /dev/nvme0n1      # Namespace details

# ── Performance Benchmarking ──────────────────────────────
# Random 4K read IOPS (database-like)
fio --name=rand-read --ioengine=libaio --iodepth=64 \
    --rw=randread --bs=4k --numjobs=4 \
    --size=100G --filename=/dev/nvme0n1 \
    --runtime=60 --time_based \
    --output-format=json | jq '.jobs[0].read.iops'

# Expected: > 500K IOPS for NVMe SSD
# Enterprise NVMe (Optane): 1M+ IOPS

# Sequential read (streaming)
fio --name=seq-read --ioengine=libaio --iodepth=32 \
    --rw=read --bs=128k --numjobs=1 \
    --size=100G --filename=/dev/nvme0n1 \
    --runtime=60 --time_based
# Expected: > 3GB/s throughput

50.5 Interview Questions

Q1: What is the difference between ZFS RAIDZ and traditional RAID-5?

Answer: Both provide single disk fault tolerance with similar space efficiency, but RAIDZ is fundamentally safer. Traditional RAID-5 has the “RAID-5 write hole”: during a power failure mid-write, parity can become inconsistent with data, causing silent corruption on rebuild. RAID-5 also suffers “URE (Unrecoverable Read Error) during rebuild” — modern large disks have ~1 URE per 10^14 bits read; rebuilding a 6TB disk means ~50TB of reads, with 50% chance of hitting a URE. RAIDZ uses ZFS’s copy-on-write — writes are atomic, no write hole. End-to-end checksums detect corruption that RAID-5 would silently propagate. RAIDZ can be RAIDZ2 or RAIDZ3 (2 or 3 parity) — strongly recommended for production.

Q2: What is the CRUSH algorithm in Ceph and why is it important?

Answer: CRUSH (Controlled Replication Under Scalable Hashing) is Ceph’s data placement algorithm. Instead of a lookup table mapping objects to OSDs (which would need updating as the cluster changes), CRUSH is a pure mathematical function: CRUSH(object_id, cluster_map) → [osd.1, osd.7, osd.23]. Given the object ID and current cluster topology, it deterministically computes where replicas should live. Importance: (1) Scales to thousands of OSDs — no central lookup table bottleneck. (2) Failure domain awareness — CRUSH knows the rack topology and places replicas across different racks/hosts. (3) Minimal rebalancing — adding/removing OSDs only remaps the minimum number of objects (not a full shuffle).

50.6 Summary

Technology	Use Case
LVM	Flexible local storage, snapshots, online resize
ZFS	Databases, NAS, data integrity critical workloads
Ceph	Distributed storage for cloud, Kubernetes PVCs
NVMe	High-IOPS databases, latency-sensitive workloads
RAIDZ2	Better than RAID-6, ZFS native
Ceph RBD	Kubernetes persistent volumes

Next Chapter: Chapter 51: Real Production Outages & Root Cause Analysis

Prerequisites

Chapter 8 (Filesystem Internals).

Advantages & Disadvantages

Advantages: ZFS prevents silent data corruption, Ceph provides infinite scalable storage. Disadvantages: High RAM requirements (ZFS ARC), high operational complexity (Ceph).

Common Mistakes

Using hardware RAID with ZFS — ZFS needs direct access to the disks to perform self-healing and checksumming.
Not dedicating enough RAM to the ZFS ARC (Adaptive Replacement Cache).
Creating an LVM snapshot and leaving it indefinitely — snapshots degrade performance and eventually fill up the volume group.

Troubleshooting

Symptom	Cause	Diagnosis	Fix
ZFS pool degraded	A disk failed or checksums don’t match	`zpool status`	Replace the faulty disk and run ‘zpool replace’; then ‘zpool scrub’
LVM volume group full	Snapshot grew too large or logical volumes took all space	`vgs; lvs`	Extend the volume group by adding a new physical volume (vgextend), or delete old snapshots

Hands-on Lab

Objective: Work with LVM.

# 1. View Physical Volumes, Volume Groups, and Logical Volumes
sudo pvs
sudo vgs
sudo lvs

# 2. View ZFS pools (if installed)
# sudo zpool status

Exercises

Create a loopback device (losetup), format it as an LVM Physical Volume, create a Volume Group, and carve out a Logical Volume.
Research Ceph architecture. Explain the role of OSDs (Object Storage Daemons) and Monitors.

Revision Notes

LVM provides abstraction over physical disks, allowing dynamic resizing and snapshots.
ZFS is a combined file system and logical volume manager with built-in data integrity (checksumming), copy-on-write, and compression.
Ceph is a distributed storage system (Block, Object, and File) designed for massive scale and self-healing.
NVMe bypasses the legacy SCSI stack, connecting directly to the PCIe bus for massive parallel I/O.