System_recovery
Chapter 45: System Recovery
Section titled “Chapter 45: System Recovery”Linux System Recovery Techniques and Procedures
Section titled “Linux System Recovery Techniques and Procedures”45.1 System Recovery Fundamentals
Section titled “45.1 System Recovery Fundamentals”Understanding System Recovery
Section titled “Understanding System Recovery”System recovery is the process of restoring a Linux system to a functional state after a failure. This can range from simple file recovery to complete system reconstruction.
SYSTEM RECOVERY SCENARIOS+------------------------------------------------------------------+| || ┌─────────────────────────────────────────────────────────┐ │| │ RECOVERY DIFFICULTY PYRAMID │ │| │ │ │| │ ┌─────────┐ │ │| │ │ Complete│ │ │| │ │ System │ │ │| │ │ Failure │ │ │| │ └────┬────┘ │ │ │ │ │ │ │ ┌────┴────┐ │ │| │ │ Kernel │ │ │| │ │ Panic │ │ │| │ └────┬────┘ │ │| │ │ │ │| │ ┌────┴────┐ │ │| │ │ Boot │ │ │| │ │ Failure │ │ │| │ └────┬────┘ │ │| │ │ │ │| │ ┌────┴────┐ │ │| │ │ Service │ │ │| │ │ Failure │ │ │| │ └────┬────┘ │ │| │ │ │ │| │ ┌────┴────┐ │ │| │ │ File │ │ │| │ │ Recovery│ │ │| │ └─────────┘ │ │| │ │ │| └─────────────────────────────────────────────────────────┘ │| |+------------------------------------------------------------------+Recovery Planning
Section titled “Recovery Planning” RECOVERY PREPARATION CHECKLIST+------------------------------------------------------------------+| || ┌─────────────────────────────────────────────────────────────┐ │| │ BEFORE FAILURE - PREPARATION │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Document system configuration │ │| │ □ Maintain current backups │ │| │ □ Keep rescue media ready │ │| │ □ Record partition layout │ │| │ □ Document network configuration │ │| │ □ Keep kernel and initramfs backups │ │| │ □ Maintain user list and passwords │ │| │ □ Document installed packages │ │| └─────────────────────────────────────────────────────────────┘ │| || ┌─────────────────────────────────────────────────────────────┐ │| │ DURING FAILURE - ASSESSMENT │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Identify scope of damage │ │| │ □ Determine backup availability │ │| │ □ Evaluate recovery options │ │| │ □ Plan recovery procedure │ │| │ □ Notify stakeholders │ │| └─────────────────────────────────────────────────────────────┘ │| || ┌─────────────────────────────────────────────────────────────┐ │| │ AFTER RECOVERY - VERIFICATION │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Test critical services │ │| │ □ Verify data integrity │ │| │ □ Update documentation │ │| │ □ Review incident │ │| │ □ Update recovery procedures │ │| └─────────────────────────────────────────────────────────────┘ │| |+------------------------------------------------------------------+45.2 Boot Recovery
Section titled “45.2 Boot Recovery”Recovery from Boot Failure
Section titled “Recovery from Boot Failure” BOOT RECOVERY PROCESS+------------------------------------------------------------------+| || BOOT FAILURE RECOVERY STEPS || ───────────── ─────────────── || || ┌────────────────┐ ┌──────────────────────────┐ │| │ GRUB Menu │ │ 1. Select recovery mode │ │| │ Not Appearing │──────→ │ from boot menu │ │| └────────────────┘ │ 2. Try 'resume' or │ ││ │ 'nomodeset' │ ││ │ 3. Check video drivers │ │| └──────────────────────────┘ │| || ┌────────────────┐ ┌──────────────────────────┐ │| │ Kernel Panic │ │ 1. Boot from rescue │ │| │ │──────→ │ media │ │| └────────────────┘ │ 2. Check kernel logs │ │| │ 3. Reinstall kernel │ │| └──────────────────────────┘ │| || ┌────────────────┐ ┌──────────────────────────┐ │| │ Initrd Error │ │ 1. Boot from rescue │ │| │ │──────→ │ media │ │| └────────────────┘ │ 2. Rebuild initramfs │ │| │ 3. Check disk drivers │ │| └──────────────────────────┘ │| || ┌────────────────┐ ┌──────────────────────────┐ │| │ File System │ │ 1. Boot from rescue │ │| │ Errors │──────→ │ media │ │| └────────────────┘ │ 2. fsck the filesystem │ │| │ 3. Fix fstab │ │| └──────────────────────────┘ │| |+------------------------------------------------------------------+Boot Recovery Commands
Section titled “Boot Recovery Commands”# =============================================================================# RECOVERY FROM GRUB SHELL# =============================================================================
# If grub prompt appears, boot manually:grub> ls# List partitions
grub> ls (hd0,msdos1)/# List files in partition
grub> set root='(hd0,msdos1)'grub> linux /boot/vmlinuz-$(uname -r) root=/dev/sda1grub> initrd /boot/initramfs-$(uname -r).imggrub> boot
# =============================================================================# RECOVERY MODE BOOT# =============================================================================
# From GRUB menu:# 1. Select "Advanced options"# 2. Choose "Recovery mode"# 3. Select "root" for root shell
# In recovery shell:# Remount root as writablemount -o remount,rw /
# Check filesystemfsck -f /dev/sda1
# Fix fstabnano /etc/fstab
# Rebuild initramfsupdate-initramfs -u
# Reinstall kernelapt-get install --reinstall linux-image-$(uname -r)
# Update GRUBupdate-grub
# =============================================================================# USING RESCUE MEDIA# =============================================================================
# Boot from live CD/USB# Then mount the system:
# Mount root partitionmount /dev/sda1 /mnt
# Mount other necessary partitionsmount --bind /dev /mnt/devmount --bind /proc /mnt/procmount --bind /sys /mnt/sys
# Chroot into systemchroot /mnt
# Now you can:# - Reinstall kernel# - Fix GRUB# - Repair packages# - Reset passwordsFixing Common Boot Issues
Section titled “Fixing Common Boot Issues”# =============================================================================# FIXING GRUB# =============================================================================
# Reinstall GRUB (from rescue media)grub-install /dev/sda
# Update GRUB configurationupdate-grub
# Reinstall GRUB2grub-install --recheck /dev/sda
# =============================================================================# FIXING KERNEL ISSUES# =============================================================================
# List installed kernelsdpkg -l | grep linux-image
# Remove broken kernelapt-get remove --purge linux-image-5.4.0-xx-generic
# Install new kernelapt-get install linux-image-generic
# Rebuild initramfsmkinitramfs -o /boot/initrd.img-$(uname -r) $(uname -r)
# =============================================================================# FIXING BOOT LOADER ISSUES# =============================================================================
# Fix bootloader configurationboot-repair
# Or manually:# Edit /etc/default/grubnano /etc/default/grub
# Common settings:# GRUB_DEFAULT=0# GRUB_TIMEOUT=10# GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
# Then updateupdate-grub45.3 File System Recovery
Section titled “45.3 File System Recovery”Using fsck for File System Repair
Section titled “Using fsck for File System Repair” FSCK RECOVERY PROCESS+------------------------------------------------------------------+| || ┌─────────────────────────────────────────────────────────┐ ││ │ FSCK FLOWCHART │ ││ │ │ ││ │ START │ ││ │ │ │ ││ │ ▼ │ ││ │ ┌─────────────────────┐ │ ││ │ │ Unmount or boot │ │ ││ │ │ to single user │ │ ││ │ └──────────┬──────────┘ │ ││ │ │ │ ││ │ ▼ │ ││ │ ┌─────────────────────┐ │ ││ │ │ Run fsck -f /dev/ │ │ ││ │ │ sdX │ │ ││ │ └──────────┬──────────┘ │ ││ │ │ │ ││ │ ▼ │ ││ │ ┌─────────────────────┐ │ ││ │ │ Errors found? │ │ ││ │ └──────────┬──────────┘ │ ││ │ YES │ NO │ ││ │ │ │ │ │ ││ │ ▼ │ ▼ │ ││ │ ┌─────────────┐ │ ┌─────────────┐ │ ││ │ │ Interactive │ │ │ Complete │ │ ││ │ │ or auto fix │ │ │ ✓ │ │ ││ │ └─────────────┘ │ └─────────────┘ │ ││ │ │ │ │ ││ │ ▼ │ │ ││ │ ┌─────────────┐ │ │ ││ │ │ Reboot and │ │ │ ││ │ │ check again │─────┘ │ ││ │ └─────────────┘ │ │| │ │ │ │| └───────────────────────┼─────────────────────────────────┘ │| ▼ || END || |+------------------------------------------------------------------+fsck Commands and Options
Section titled “fsck Commands and Options”# =============================================================================# FSCK BASIC USAGE# =============================================================================
# Check filesystem (unmount first!)umount /dev/sda1fsck /dev/sda1
# Check with automatic repairfsck -a /dev/sda1
# Check and ask for each fixfsck -r /dev/sda1
# Force check even if cleanfsck -f /dev/sda1
# Verbose outputfsck -v /dev/sda1
# Check specific filesystem typefsck -t ext4 /dev/sda1
# =============================================================================# FSCK FOR DIFFERENT FILESYSTEMS# =============================================================================
# ext4fsck.ext4 -p /dev/sda1 # Auto repairfsck.ext4 -f /dev/sda1 # Force check
# XFSxfs_repair /dev/sda1xfs_repair -n /dev/sda1 # Dry run
# BTRFSbtrfs check --repair /dev/sda1btrfs check --readonly /dev/sda1 # Check without repair
# =============================================================================# RECOVERING DELETED FILES (with extundelete)# =============================================================================
# Install extundeleteapt-get install extundelete
# List deleted filesextundelete /dev/sda1 --inode 2
# Restore all deleted filesextundelete /dev/sda1 --restore-all
# Restore specific fileextundelete /dev/sda1 --restore-file /path/to/deleted/file
# Restore files from specific directoryextundelete /dev/sda1 --restore-directory /homeTestDisk for Partition Recovery
Section titled “TestDisk for Partition Recovery”# =============================================================================# INSTALL TESTDISK# =============================================================================
apt-get install testdisk
# =============================================================================# RECOVER PARTITIONS# =============================================================================
# Run testdisktestdisk
# Steps in testdisk:# 1. Select disk# 2. Select partition table type (usually Intel/EFI)# 3. Select "Analyse" to search for partitions# 4. Select "Search!" to search deeper# 5. Select found partition and press "P" to list files# 6. Select files to recover# 7. Press "Write" to write partition table
# =============================================================================# PHOTOREC FOR FILE RECOVERY# =============================================================================
# Run photorecphotorec
# Steps:# 1. Select disk# 2. Select partition# 3. Select filesystem type (Other for ext4)# 4. Select "Free" for free space or "Whole" for entire partition# 5. Select destination directory# 6. Wait for recovery to complete45.4 Password Recovery
Section titled “45.4 Password Recovery”Resetting Root Password
Section titled “Resetting Root Password” PASSWORD RECOVERY METHODS+==================================================================+| || METHOD 1: SINGLE USER MODE (GRUB) || ───────────────────────────────────── || || 1. Reboot system || 2. Press 'e' at GRUB menu to edit || 3. Find line starting with 'linux' || 4. Add 'init=/bin/bash' at end || 5. Press Ctrl+X to boot || 6. Mount root: mount -o remount,rw / || 7. Reset password: passwd root || 8. Reboot: exec /sbin/init || || METHOD 2: RESCUE MEDIA || ────────────────────────── || || 1. Boot from Live CD/USB || 2. Mount system partition || 3. chroot into system || 4. Reset password: passwd root || || METHOD 3: SYSTEMD (Modern Systems) || ─────────────────────────────────── || || 1. At GRUB, edit boot entry || 2. Add to linux line: systemd.unit=rescue.target || 3. Boot and run: passwd root || |+==================================================================+Password Recovery Commands
Section titled “Password Recovery Commands”# =============================================================================# RESET PASSWORD FROM SINGLE USER MODE# =============================================================================
# From GRUB edit (add to linux line):rw init=/bin/bash
# After boot:mount -o remount,rw /passwd root# Or for specific user:passwd username
# If SELinux is enabled:load_policy -ipasswd root
# Reboot:exec /sinst
# =============================================================================# RESET PASSWORD WITH CHROOT (FROM LIVE MEDIA)# =============================================================================
# Boot from live media# Mount root partitionmount /dev/sda1 /mnt
# Mount required filesystemsmount --bind /dev /mnt/devmount --bind /proc /mnt/procmount --bind /sys /mnt/sys
# Chrootchroot /mnt
# Reset passwordpasswd root
# Exit and rebootexitreboot
# =============================================================================# BYPASS PAM AUTHENTICATION (EMERGENCY)# =============================================================================
# Edit PAM configuration temporarily# In chroot:nano /etc/pam.d/common-auth
# Comment out auth lines (USE WITH CAUTION!)# auth required pam_permit.so45.5 System Reconstruction
Section titled “45.5 System Reconstruction”Complete System Reinstall Procedure
Section titled “Complete System Reinstall Procedure”# =============================================================================# STEPS FOR COMPLETE SYSTEM RECONSTRUCTION# =============================================================================
# 1. Document current system (if possible)# From working system or backup:dpkg --get-selections > /tmp/packages.txtdpkg -l > /tmp/packages_list.txtcat /etc/fstab > /tmp/fstab.txtcat /etc/passwd > /tmp/passwd.txtcat /etc/group > /tmp/group.txt
# 2. List running servicessystemctl list-unit-files > /tmp/services.txt
# 3. Document network configurationip addr show > /tmp/network.txtcat /etc/network/interfaces > /tmp/interfaces.txt
# 4. Reinstall base system# Using apt:apt-get install --reinstall coreutils bash glibc
# 5. Reinstall all packagesdpkg --set-selections < /tmp/packages.txtapt-get dselect-upgrade
# 6. Restore configurations# Restore from backup:cp /backup/etc/passwd /etc/passwdcp /backup/etc/group /etc/groupcp /backup/etc/shadow /etc/shadow # If available
# 7. Restore servicessystemctl enable nginxsystemctl enable mysql
# 8. Verify systemsystemctl statusrebootRecovering from Kernel Panic
Section titled “Recovering from Kernel Panic” KERNEL PANIC RECOVERY+==================================================================+| || COMMON CAUSES AND SOLUTIONS || ───────────────────────────────── || || ┌───────────────────┬─────────────────────────────────────┐ │| │ CAUSE │ SOLUTION │ │| ├───────────────────┼─────────────────────────────────────┤ │| │ Hardware issues │ - Run memtest │ │| │ │ - Check hardware diagnostics │ │| ├───────────────────┼─────────────────────────────────────┤ │| │ Driver problems │ - Boot with nomodeset │ │| │ │ - Remove problematic drivers │ │| ├───────────────────┼─────────────────────────────────────┤ │| │ Out of memory │ - Add kernel parameter: │ │| │ │ mem=xxxxM │ │| ├───────────────────┼─────────────────────────────────────┤ │| │ Filesystem errors │ - Boot rescue mode │ │| │ │ - Run fsck │ │| ├───────────────────┼─────────────────────────────────────┤ │| │ Init process │ - Check /sbin/init │ │| │ failure │ - Rebuild initramfs │ │| └───────────────────┴─────────────────────────────────────┘ │| |+==================================================================+45.6 Recovery Best Practices
Section titled “45.6 Recovery Best Practices”Recovery Procedure Checklist
Section titled “Recovery Procedure Checklist” SYSTEM RECOVERY CHECKLIST+==================================================================+| || ┌─────────────────────────────────────────────────────────────┐ │| │ INITIAL ASSESSMENT │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Identify the scope of the problem │ │| │ □ Determine if data is at risk │ │| │ □ Check backup availability │ │| │ □ Evaluate time required for each recovery option │ │| │ □ Get approval if needed │ │| └─────────────────────────────────────────────────────────────┘ │| || ┌─────────────────────────────────────────────────────────────┐ │| │ BOOT RECOVERY │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Try recovery mode boot │ │| │ □ Check GRUB configuration │ │| │ □ Verify kernel and initramfs │ │| │ □ Test with fallback kernel │ │| │ □ Check bootloader installation │ │| └─────────────────────────────────────────────────────────────┘ │| || ┌─────────────────────────────────────────────────────────────┐ │| │ FILESYSTEM RECOVERY │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Run filesystem check │ │| │ □ Recover deleted files │ │| │ □ Restore from backup │ │| │ □ Verify data integrity │ │| │ □ Check disk SMART status │ │| └─────────────────────────────────────────────────────────────┘ │| || ┌─────────────────────────────────────────────────────────────┐ │| │ POST-RECOVERY │ │| ├─────────────────────────────────────────────────────────────┤ │| │ □ Verify all services start │ │| │ □ Test critical functionality │ │| │ □ Update monitoring │ │| │ □ Document lessons learned │ │| │ □ Schedule follow-up review │ │| └─────────────────────────────────────────────────────────────┘ │| |+==================================================================+Emergency Contacts and Resources
Section titled “Emergency Contacts and Resources” RECOVERY EMERGENCY RESOURCES+==================================================================+| || LINKS: │| ───── │| - Arch Wiki: https://wiki.archlinux.org │| - Red Hat KB: https://access.redhat.com │| - Ubuntu Wiki: https://help.ubuntu.com │| || EMERGENCY TOOLS: │| ─────────────── │| - SystemRescue: https://systemrescue.org │| - GParted Live: https://gparted.org │| - Knoppix: https://www.knoppix.org │| || DOCUMENTATION TO KEEP: │| ─────────────────────── │| - Partition layout │| - Network configuration │| - Installed packages list │| - Service configuration │| - Backup strategy and locations │| |+==================================================================+End of Chapter 45: System Recovery