Disaster Recovery: RPO, RTO, Backup Strategies & Testing

Chapter 39: Disaster Recovery: RPO, RTO, Backup Strategies & Testing

Learning Objectives

• Define and calculate RPO, RTO, MTTR, and MTBF for your services
• Design backup and recovery strategies for databases and filesystems
• Implement DR runbooks and test them regularly
• Understand active-passive vs active-active DR patterns

---

What is this? (Beginner On-Ramp)

Disaster Recovery (DR) is the emergency plan for when High Availability fails and a catastrophic event (like a datacenter fire or ransomware) wipes out the system. It’s the process of bringing the entire business back online from secure backups.

39.1 DR Fundamentals

    Key DR Metrics
    ──────────────────────────────────────────────────────

    RPO (Recovery Point Objective):
    "How much data can we afford to lose?"
    Time between last backup and the disaster

    Example: Backups every 4 hours → RPO = 4 hours
    If disaster at 3:50pm, last backup at 12:00pm → lose 3:50 of data

    RTO (Recovery Time Objective):
    "How long can we afford to be down?"
    Time from disaster to full service restoration

    Example: RTO = 30 minutes
    You have 30 minutes to restore service before SLA breach

    MTBF (Mean Time Between Failures):
    Average time system runs between failures
    Higher = more reliable

    MTTR (Mean Time To Recover):
    Average time to restore service after failure
    Lower = better

    Availability = MTBF / (MTBF + MTTR)

    DR Tier Examples:
    ──────────────────
    Tier 1 (Mission Critical): RPO 0s, RTO 0s — active-active, no downtime
    Tier 2 (Business Critical): RPO 1h, RTO 4h — warm standby
    Tier 3 (Important):         RPO 4h, RTO 24h — cold standby
    Tier 4 (Non-critical):      RPO 24h, RTO 72h — backup restore

---

39.2 DR Architecture Patterns

    Cold Standby (Backup & Restore)
    ─────────────────────────────────
    Primary ─── Backups ──► S3/Object Storage

    Recovery: Restore backup to new server
    RPO: Hours (backup interval)
    RTO: Hours (restore time)
    Cost: Low
    Use: Non-critical data, low budget

    Warm Standby (Pilot Light)
    ──────────────────────────
    Primary ─── Replication ──► Standby (minimal, scaled down)

    Recovery: Scale up standby, update DNS
    RPO: Minutes (replication lag)
    RTO: 30-60 minutes (scale + DNS TTL)
    Cost: Medium
    Use: Most business applications

    Hot Standby (Active-Passive)
    ─────────────────────────────
    Primary ─── Sync Replication ──► Standby (full size, waiting)

    Recovery: Promote standby, failover DNS
    RPO: Seconds (sync lag)
    RTO: Minutes
    Cost: High (full duplicate infra)
    Use: Critical services

    Active-Active (Multi-Region)
    ──────────────────────────────
    Region A ◄──── Traffic split ────► Region B
         ▲          (50%/50%)              ▲
         └────── Bidirectional sync ───────┘

    Recovery: Remove failed region from traffic
    RPO: 0 (if sync is zero-lag)
    RTO: 0 (traffic auto-reroutes)
    Cost: Very High (2x infra + complex sync)
    Use: Mission critical (payment, auth)

---

39.3 Database Backup Strategies

# ── PostgreSQL: Full + WAL Archiving ─────────────────────
# Full base backup (pg_basebackup)
pg_basebackup -h localhost -U postgres \
    -D /backups/$(date +%Y%m%d) \
    --format=tar --compress=9 \
    --checkpoint=fast \
    --progress

# WAL archiving (continuous backup between full backups)
# postgresql.conf:
# wal_level = replica
# archive_mode = on
# archive_command = 'aws s3 cp %p s3://backups/wal/%f'

# Point-in-Time Recovery (PITR)
# Restore base backup, then replay WALs to specific moment
# recovery.conf:
# restore_command = 'aws s3 cp s3://backups/wal/%f %p'
# recovery_target_time = '2024-01-15 14:30:00'

# ── Barman: PostgreSQL Backup Manager ────────────────────
# /etc/barman.d/prod.conf
[prod]
description = Production PostgreSQL
conninfo = host=db.internal user=barman dbname=postgres
backup_method = postgres
streaming_conninfo = host=db.internal user=streaming_barman
streaming_archiver = on
slot_name = barman
retention_policy = RECOVERY WINDOW OF 30 DAYS

# Backup
barman backup prod

# List backups
barman list-backup prod

# Restore to point in time
barman recover prod 20240115T120000 /var/lib/postgresql/data \
    --target-time "2024-01-15 14:30:00" \
    --remote-ssh-command "ssh postgres@db-new.internal"

# ── MySQL: Percona XtraBackup ─────────────────────────────
# Full backup (hot, no lock)
xtrabackup --backup --target-dir=/backups/full/

# Incremental backup
xtrabackup --backup --target-dir=/backups/inc/$(date +%Y%m%d) \
    --incremental-basedir=/backups/full/

# Restore
xtrabackup --prepare --target-dir=/backups/full/
xtrabackup --copy-back --target-dir=/backups/full/

# ── Volume Snapshots (Cloud) ──────────────────────────────
# AWS EBS snapshot (consistent for databases)
# 1. Quiesce writes (or use filesystem freeze)
aws ec2 create-snapshot \
    --volume-id vol-xxx \
    --description "prod-db $(date +%Y%m%d-%H%M)"

# Or freeze filesystem temporarily
fsfreeze -f /var/lib/postgresql
aws ec2 create-snapshot --volume-id vol-xxx
fsfreeze -u /var/lib/postgresql

---

39.4 Backup Verification & DR Testing

    Backup Testing (critical — most teams skip this!)
    ──────────────────────────────────────────────────

    Untested backups are not backups.

    Monthly DR test process:
    1. Restore latest backup to isolated environment
    2. Run application against restored data
    3. Verify data integrity (row counts, checksums)
    4. Measure actual RTO (how long did restore take?)
    5. Compare against RTO target
    6. Document findings, fix gaps

    Reality check:
    "Our RTO is 2 hours" → Test it!
    Common discovery: restore takes 6 hours (database is 3x bigger than estimated)

weekly-restore-test.sh

#!/bin/bash
set -euo pipefail

BACKUP_DATE=$(date -d '1 day ago' +%Y%m%d)
TEST_HOST="db-restore-test.internal"
START_TIME=$(date +%s)

echo "=== DR Test: Restoring backup from $BACKUP_DATE ==="

# Download backup from S3
aws s3 sync s3://backups/postgres/$BACKUP_DATE /tmp/restore/

# Restore on test host
pg_restore -h $TEST_HOST -U postgres -d testdb /tmp/restore/dump.sql

# Verify integrity
PROD_COUNT=$(psql -h prod-db -c "SELECT count(*) FROM orders" -t | tr -d ' ')
TEST_COUNT=$(psql -h $TEST_HOST -c "SELECT count(*) FROM orders" -t | tr -d ' ')

if [ "$PROD_COUNT" != "$TEST_COUNT" ]; then
    echo "FAIL: Row count mismatch! prod=$PROD_COUNT test=$TEST_COUNT"
    exit 1
fi

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
echo "SUCCESS: Restore completed in ${DURATION}s ($(($DURATION/60)) minutes)"
echo "Row count verified: $TEST_COUNT rows"

# Alert if RTO exceeded
RTO_SECONDS=7200   # 2 hours
if [ $DURATION -gt $RTO_SECONDS ]; then
    echo "WARNING: RTO exceeded! Actual: ${DURATION}s Target: ${RTO_SECONDS}s"
fi

---

39.5 DR Runbook Template

# Runbook: Database Disaster Recovery

## Trigger Conditions
- Primary database unreachable for > 5 minutes
- Data corruption detected
- Datacenter failure

## RTO: 30 minutes | RPO: 5 minutes (streaming replication)

## Step 1: Confirm disaster (5 minutes)
1. Verify primary is actually down (not network partition):
   `psql -h db-primary.internal -c "SELECT 1"`
2. Check replication lag on standby:
   `psql -h db-standby.internal -c "SELECT now() - pg_last_xact_replay_timestamp()"`
3. Page database team: @dba-oncall

## Step 2: Failover to standby (5 minutes)
1. Promote standby:
   `pg_ctl promote -D /var/lib/postgresql/data`
   OR: `touch /tmp/postgresql.trigger.5432`
2. Verify standby accepted writes:
   `psql -h db-standby.internal -c "SELECT pg_is_in_recovery()"`
   → Should return: f (false = primary mode)

## Step 3: Update connection strings (10 minutes)
1. Update Route53: db.internal → db-standby.internal (TTL 60s)
2. Rotate Vault DB secrets to point to new primary
3. Rolling restart app servers (pick up new connection strings)

## Step 4: Verify (5 minutes)
1. Application health checks green
2. No errors in application logs
3. Confirm orders/transactions flowing normally

## Step 5: Notify (5 minutes)
1. Update status page
2. Post in #incidents Slack
3. Create post-mortem ticket

## Post-Recovery (next day)
1. Rebuild replica from new primary
2. Re-establish replication
3. Restore VIP to original configuration if desired

---

39.6 Interview Questions

Q1: What is the difference between RPO and RTO? Give a practical example.

Answer: RPO (Recovery Point Objective) is the maximum acceptable data loss — how far back in time you can go if disaster strikes. If your RPO is 1 hour, you back up every hour, and losing 59 minutes of transactions is acceptable. RTO (Recovery Time Objective) is the maximum acceptable downtime — how long you can be non-operational. If your RTO is 30 minutes, you must be able to restore service within 30 minutes of disaster. Practical example: A payment system has RPO=0 (no data loss) and RTO=0 (no downtime) — requires active-active multi-region with synchronous replication. A reporting system might have RPO=24h and RTO=8h — daily backups and manual restore process is fine.

Q2: Why must you regularly test your backups and disaster recovery plan?

Answer: Because untested backups frequently fail in ways you don’t discover until you need them: (1) Corrupt backups: Backup process seemed to run but created corrupted files. (2) Missing dependencies: Backup restored but application can’t start because config files, certificates, or dependent services weren’t backed up. (3) Wrong RTO: “RTO is 2 hours” but the database is now 10x bigger — restore actually takes 8 hours. (4) Runbook is outdated: Infrastructure changed but runbook wasn’t updated — the failover steps reference servers that no longer exist. Test quarterly at minimum. Netflix does Game Days — intentional failure injection. Amazon does Wheel of Misfortune exercises.

---

39.7 Summary

Concept	Value
RPO	Max acceptable data loss
RTO	Max acceptable downtime
Cold standby	High RPO/RTO, low cost
Warm standby	Medium RPO/RTO, medium cost
Hot standby	Low RPO/RTO, high cost
Active-active	Zero RPO/RTO, very high cost
Test frequency	At minimum quarterly

---

Next Chapter: Chapter 40: Load Balancing Strategies & Algorithms

Prerequisites

Chapter 38 (High Availability).

---

Advantages & Disadvantages

Advantages: Guarantees business survival after a catastrophic event (ransomware/fire). Disadvantages: Testing DR is terrifying and expensive, backups are useless if not tested.

---

Common Mistakes

• Never testing backups (Schrödinger’s Backup).
• Keeping backups in the same availability zone/datacenter as the primary data.
• Confusing HA (uptime) with DR (data recovery from catastrophic failure).

---

Troubleshooting

Symptom	Cause	Diagnosis	Fix
Restore from backup fails	Corrupted backup or mismatched software version	Review restore logs; test backups regularly	Maintain strict backup testing schedules
RTO exceeded during outage	Manual recovery steps take too long	Review incident timeline	Automate recovery processes (Infrastructure as Code)

---

Hands-on Lab

Objective: Understand RPO and RTO.

# Conceptual lab:
# Review your database backup cron job.
# If it runs daily at midnight, what is your RPO? (Answer: up to 24 hours).
# If a restore takes 4 hours to complete, what is your minimum RTO? (Answer: 4 hours).

---

Exercises

1. Write a script that dumps a PostgreSQL database, compresses it, encrypts it, and uploads it to an S3 bucket.
2. Design a DR plan for a simple web application. Define the RTO, RPO, and the exact steps to failover to a secondary region.

---

Revision Notes

• RPO (Recovery Point Objective): How much data loss is acceptable (time).
• RTO (Recovery Time Objective): How long recovery should take.
• HA is about preventing downtime; DR is about recovering from catastrophic failure.
• Backups must be tested, encrypted, and stored off-site (3-2-1 rule).

---

Further Reading

• AWS Disaster Recovery Whitepaper
• Google SRE Book - Disaster Recovery

---

Related Chapters

• Chapter 38 — High Availability

---

Last Updated: July 2026