Ha_concepts

Chapter 61: Backup and Disaster Recovery

Overview

Backup and disaster recovery are critical for any system administrator. This chapter covers various backup strategies, tools, disaster recovery planning, and testing procedures essential for production environments.

61.1 Backup Strategies

Types of Backups

                          Backup Types
+------------------------------------------------------------------+
|                                                                   |
|                    +------------------------+                     |
|                    |    Backup Types        |                     |
|                    +------------------------+                     |
|                              |                                    |
|        +---------------------+---------------------+              |
|        |                     |                     |              |
|        v                     v                     v              |
|   +---------+          +-----------+         +-----------+        |
|   |  Full   |          |Incremental|         |Differential|       |
|   +---------+          +-----------+         +-----------+        |
|        |                     |                     |              |
|        v                     v                     v              |
|   +---------+          +-----------+         +-----------+        |
|   |All Data |          |Changes    |         |Changes   |        |
|   |         |          |since last |         |since last |        |
|   |Copy     |          |backup     |         |full backup|        |
|   +---------+          +-----------+         +-----------+        |
|                                                                   |
+------------------------------------------------------------------+

Backup Types Explained

# Full Backup
# - Complete copy of all data
# - Largest size, longest time
# - Easiest to restore

# Incremental Backup
# - Only changes since last backup
# - Smallest size, fastest
# - Slowest to restore (need all backups)

# Differential Backup
# - Changes since last full backup
# - Medium size
# - Faster restore than incremental

The 3-2-1 Backup Rule

# 3 copies of data
# 2 different storage types
# 1 offsite copy

61.2 rsync for Backups

Basic rsync

# Local backup
rsync -av /source/ /destination/
rsync -avz /source/ destination@server:/path/    # Over network

# Options explained
-a  # Archive mode (preserves permissions, timestamps, etc.)
-v  # Verbose
-z  # Compress during transfer
-h  # Human readable
-P  # Show progress, resume partial
--delete           # Delete files not in source
--exclude='*.log'  # Exclude patterns
--dry-run          # Test without making changes

rsync Examples

# Backup home directory
rsync -avPh --delete /home/user/ /backup/user/

# Backup with exclude
rsync -av --exclude='node_modules/' --exclude='.git/' /app/ /backup/app/

# Remote backup over SSH
rsync -avz -e ssh /local/ user@remote:/backup/

# Preserve hard links
rsync -avH /source/ /destination/

# Bandwidth limit (KB/s)
rsync -avz --bwlimit=1000 /source/ /destination/

rsync Script

#!/bin/bash
SOURCE="/home"
DEST="/backup"
LOGFILE="/var/log/backup.log"
EMAIL="admin@example.com"

log() {
    echo "[$(date)] $*" | tee -a $LOGFILE
}

log "Starting backup from $SOURCE to $DEST"

# Create destination if not exists
mkdir -p "$DEST"

# Run rsync
if rsync -avH --delete \
    --exclude='.cache' \
    --exclude='.local/share/Trash' \
    "$SOURCE/" "$DEST/"; then
    log "Backup completed successfully"

    # Get backup size
    du -sh "$DEST"
else
    log "Backup FAILED"
    echo "Backup failed" | mail -s "Backup Failed" "$EMAIL"
fi

61.3 tar for Backups

Creating Backups with tar

# Create compressed archive
tar -czvf backup.tar.gz /path/to/dir
tar -cajvf backup.tar.bz2 /path/to/dir
tar -cavf backup.tar.xz /path/to/dir

# Create archive with date
tar -czvf "backup_$(date +%Y%m%d).tar.gz" /path/to/dir

# Exclude patterns
tar -czvf backup.tar.gz \
    --exclude='*.log' \
    --exclude='node_modules' \
    /path/to/dir

tar Examples

# Backup and split into parts
tar -czvf - /path | split -b 1000M - backup_part_

# Backup with preserve attributes
tar -cpzvf backup.tar.gz /path

# c = create, p = preserve, z = gzip, v = verbose

# Extract
tar -xzvf backup.tar.gz
tar -xajvf backup.tar.bz2

# Extract to specific directory
tar -xzvf backup.tar.gz -C /destination/

# List contents
tar -tzvf backup.tar.gz

61.4 Restic - Modern Backup Tool

Installing and Basic Usage

# Install restic
sudo pacman -S restic

# Initialize repository
restic init --repo /backup/repo
# Or with password file
restic init --repo /backup/repo --password-file /etc/restic_pw

# Backup
restic backup /home
restic backup /home --tag "daily"

# List backups
restic snapshots

# Restore
restic restore latest --target /restore

# Check integrity
restic check

# Remove old backups
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6

Restic Remote Backends

# S3 backend
restic -r s3:https://s3.amazonaws.com/bucket/repo

# Backblaze B2
restic -r b2:bucketname:reponame

# SFTP
restic -r sftp:user@host:/path/to/repo

# REST server
restic -r rest:https://backup.example.com:8000/repo

61.5 duplicity - Encrypted Backups

# Install duplicity
sudo pacman -S duplicity

# Backup to local file
duplicity /home file:///backup

# Backup to remote (SFTP)
duplicity /home sftp://user@host//backup

# Backup to AWS S3
duplicity /home s3://bucketname/prefix

# Full backup (full and incremental default)
duplicity full /home sftp://user@host//backup

# Incremental backup
duplicity /home sftp://user@host//backup

# Restore
duplicity sftp://user@host//backup /restore

# List files in backup
duplicity list-current-files sftp://user@host//backup

# Verify backup
duplicity verify sftp://user@host//backup /home

61.6 Database Backups

PostgreSQL Backup

# pg_dump - single database
pg_dump -U postgres mydb > backup.sql
pg_dump -U postgres mydb | gzip > backup.sql.gz

# pg_dumpall - all databases
pg_dumpall -U postgres > all_databases.sql

# Custom format (for pg_restore)
pg_dump -U postgres -Fc mydb > backup.dump

# Parallel backup (large databases)
pg_dump -U postgres -Fd -j 4 mydb -f backup_dir

# Restore
psql -U postgres mydb < backup.sql
pg_restore -U postgres -d mydb backup.dump

# Point-in-time recovery (PITR)
# Requires WAL archiving enabled

MySQL/MariaDB Backup

# mysqldump - single database
mysqldump -u root -p mydb > backup.sql
mysqldump -u root -p mydb | gzip > backup.sql.gz

# All databases
mysqldump -u root -p --all-databases > all_databases.sql

# With options for production
mysqldump -u root -p \
    --single-transaction \
    --quick \
    --lock-tables=false \
    mydb > backup.sql

# Restore
mysql -u root -p mydb < backup.sql

# Binary log backup (for PITR)
mysqlbinlog mysql-bin.000001 > backup.sql

61.7 System Backup

Systemd-snapshot

# Create btrfs snapshot
sudo btrfs subvolume snapshot / /.snapshots/$(date +%Y%m%d)

# List snapshots
btrfs subvolume list /

# Restore snapshot
sudo btrfs subvolume snapshot /.snapshots/backup /

# Delete snapshot
sudo btrfs subvolume delete /.snapshots/backup

Timeshift (for Arch)

# Install timeshift
sudo pacman -S timeshift

# Create snapshot
sudo timeshift --create --comments "Before update"

# List snapshots
sudo timeshift --list

# Restore
sudo timeshift --restore

# Delete snapshot
sudo timeshift --delete --snapshot '2024-01-01_00-00'

61.8 Disaster Recovery Planning

DR Strategy Components

                     Disaster Recovery Components
+------------------------------------------------------------------+
|                                                                   |
|   Recovery              Backups              Procedures          |
|   +---------+         +---------+          +---------+          |
|   |   RTO   |         |  Local  |          |   Doc   |          |
|   |Recovery |         | Backup  |          |         |          |
|   | Time    |         +---------+          +---------+          |
|   |Objective|         | Offsite |          |Testing |          |
|   +---------+         | Backup  |          |         |          |
|   +---------+         +---------+          +---------+          |
|   |   RPO   |         | Cloud   |          |Automation|         |
|   |Recovery |         | Backup  |          |         |          |
|   | Point   |         +---------+          +---------+          |
|   |Objective|                                                     |
|   +---------+                                                     |
|                                                                   |
+------------------------------------------------------------------+

RTO and RPO

# RTO - Recovery Time Objective
# Maximum acceptable downtime
# Example: 4 hours

# RPO - Recovery Point Objective
# Maximum acceptable data loss (time)
# Example: 24 hours (daily backups = max 24h loss)

DR Document Template

# Disaster Recovery Plan

## Critical Systems
1. Web Application
2. Database
3. File Storage

## RTO/RPO
- Web Application: RTO 1 hour, RPO 1 hour
- Database: RTO 4 hours, RPO 24 hours
- File Storage: RTO 8 hours, RPO 24 hours

## Backup Locations
- Local: /backup/local
- Offsite: rsync to backup-server
- Cloud: AWS S3

## Recovery Procedures

### Web Application
1. Provision new server
2. Deploy from version control
3. Restore configuration
4. Update DNS

### Database
1. Provision new database server
2. Restore from latest backup
3. Verify data integrity
4. Update connection strings

### Testing Schedule
- Monthly: Full restore test
- Weekly: Backup verification

61.9 Backup Monitoring

Backup Verification Script

#!/bin/bash
BACKUP_DIR="/backup"
EMAIL="admin@example.com"
MAX_AGE=25  # hours

# Check backup exists and recent
last_backup=$(find "$BACKUP_DIR" -type f -mmin -$((MAX_AGE * 60)) | head -1)

if [ -z "$last_backup" ]; then
    echo "WARNING: No backup found in $MAX_AGE hours" | \
        mail -s "Backup Alert" "$EMAIL"
    exit 1
fi

# Verify backup can be extracted
if tar -tzf "$last_backup" > /dev/null 2>&1; then
    echo "Backup verification successful"
else
    echo "WARNING: Backup is corrupted" | \
        mail -s "Backup Corrupted" "$EMAIL"
    exit 1
fi

# Check backup size (should be non-empty)
size=$(stat -f%z "$last_backup" 2>/dev/null || stat -c%s "$last_backup")
if [ "$size" -lt 1000 ]; then
    echo "WARNING: Backup suspiciously small" | \
        mail -s "Backup Size Alert" "$EMAIL"
fi

61.10 Complete Backup Solution

rsnapshot Configuration

# Install rsnapshot
sudo pacman -S rsnapshot

# Main configuration file (tab-separated, not spaces!)

snapshot_root   /backup/

interval        daily   7
interval        weekly  4
interval        monthly 3

backup  /home/          localhost/
backup  /etc/           localhost/
backup  /var/lib/postgresql/    localhost/

# Exclude
backup_script   /usr/local/bin/backup_excludes.sh   excluded/

# Run backup
rsnapshot daily
rsnapshot weekly
rsnapshot monthly

Cron Jobs for Backups

# Daily at 2 AM
0 2 * * * /usr/local/bin/backup.sh

# Weekly on Sunday at 3 AM
0 3 * * 0 /usr/local/bin/backup_weekly.sh

# Monthly on 1st at 4 AM
0 4 1 * * /usr/local/bin/backup_monthly.sh

61.11 Restoring from Disaster

Recovery Procedures

# 1. Assess the situation
# - What failed?
# - How much data lost?
# - What's the impact?

# 2. Start recovery
# - Provision new hardware/VM
# - Install OS
# - Restore configuration

# 3. Restore data
# - Mount backup storage
# - Extract backups
# - Verify integrity

# 4. Verify services
# - Start services
# - Check logs
# - Test functionality

# 5. Update documentation
# - Document what happened
# - Update DR plan if needed

Emergency Boot Recovery

# Boot from rescue media
# Check filesystem
fsck -y /dev/sda1

# Mount root
mount /dev/sda1 /mnt
mount -o bind /proc /mnt/proc
mount -o bind /dev /mnt/dev
mount -o bind /sys /mnt/sys

# Chroot
chroot /mnt /bin/bash

# Reinstall bootloader
grub-install /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

# Exit chroot
exit
umount -R /mnt
reboot

Summary

In this chapter, you learned:

✅ Backup strategies (full, incremental, differential)
✅ 3-2-1 backup rule
✅ rsync for efficient backups
✅ tar for archive backups
✅ Restic for modern encrypted backups
✅ duplicity for encrypted remote backups
✅ Database backups (PostgreSQL, MySQL)
✅ System snapshots (btrfs, timeshift)
✅ Disaster recovery planning
✅ RTO/RPO concepts
✅ Backup monitoring and verification
✅ Recovery procedures

Next Chapter

Chapter 16: SSH Security and Hardening

Last Updated: February 2026