File_ops

Chapter 18: File Operations in Bash

Overview

This chapter covers file operations in Bash - a critical skill for DevOps engineers and system administrators. File operations are the backbone of automation scripts, configuration management, log processing, and backup solutions.

---

Understanding Files in Linux

What is a File?

In Linux, everything is a file:

• Regular files (-)
• Directories (d)
• Symbolic links (l)
• Character devices (c)
• Block devices (b)
• Named pipes (p)
• Sockets (s)

File Descriptors

Linux uses file descriptors (FD) to access files:

FD Number	Name	Default Destination
0	stdin	Keyboard
1	stdout	Terminal
2	stderr	Terminal

---

Reading Files

Method 1: Using cat

The simplest way to read a file:

# Display entire file
cat /etc/hostname

# Display with line numbers
cat -n /etc/hostname

# Display with line numbers (non-empty lines)
cat -b /etc/hostname

# Show $ at end of lines (useful for debugging whitespace)
cat -A /etc/hostname

Real-world DevOps Example:

# View configuration file
cat /etc/nginx/nginx.conf

# Check application config
cat /var/www/html/app/config.php

Method 2: Using less (Recommended for Large Files)

# Open file for viewing (large files)
less /var/log/syslog

# Common less commands:
# /search    - Search forward
# ?search    - Search backward
# n          - Next match
# N          - Previous match
# g          - Go to beginning
# G          - Go to end
# q          - Quit
# -N         - Show line numbers

# Practical examples
less +G /var/log/messages     # Start at end of file
less -N /var/log/syslog       # Show line numbers
less -S /var/log/nginx.log    # Don't wrap long lines

Method 3: Using head

Display the first lines of a file:

# First 10 lines (default)
head /var/log/syslog

# First N lines
head -n 20 /var/log/syslog

# First N bytes
head -c 1KB /var/log/syslog

# DevOps examples
head -1 /etc/passwd           # Show first user
head -n 100 /var/log/nginx/access.log  # First 100 requests

Method 4: Using tail

Display the last lines of a file:

# Last 10 lines (default)
tail /var/log/syslog

# Last N lines
tail -n 20 /var/log/syslog

# Last N bytes
tail -c 500 /var/log/syslog

# Follow file in real-time (critical for monitoring)
tail -f /var/log/syslog
tail -f /var/log/nginx/error.log

# Follow with specific lines
tail -f -n 100 /var/log/app.log

# DevOps Examples
tail -f /var/log/auth.log     # Monitor login attempts
tail -f /var/log/nginx/access.log  # Monitor live traffic
tail -n 50 /var/log/docker.log    # Recent Docker events

Method 5: Using awk for Selective Reading

# Print specific columns
awk '{print $1, $5}' /var/log/nginx/access.log

# Print lines matching pattern
awk '/ERROR/ {print}' /var/log/app.log

# Print specific line numbers
awk 'NR==10 {print}' /var/log/syslog

# Print first and last field
awk '{print $1, $NF}' /etc/passwd

Method 6: Using sed for Line Selection

# Print specific line
sed -n '5p' /etc/passwd

# Print range of lines
sed -n '5,10p' /etc/passwd

# Print every Nth line
sed -n '~5p' /var/log/syslog  # Every 5th line

---

Writing and Appending to Files

Redirect Output to New File

# Create new file (overwrites existing)
echo "Hello World" > /tmp/output.txt
cat /etc/hostname > /tmp/hostname.txt

# Using printf for more control
printf "%s\n" "Line 1" "Line 2" "Line 3" > /tmp/output.txt

Append to Existing File

# Append to file (creates if not exists)
echo "New line" >> /tmp/output.txt

# Append multiple lines
cat >> /tmp/log.txt <<EOF
$(date) - Application started
$(date) - Health check passed
$(date) - Backup completed
EOF

Writing with tee

The tee command writes to both file and stdout:

# Write and display
echo "Logging this" | tee /var/log/app.log

# Append and display
echo "New entry" | tee -a /var/log/app.log

# Tee to multiple files
echo "Data" | tee file1.txt file2.txt file3.txt

# Suppress stdout
echo "Silent log" | tee -s /var/log/app.log

Real-world DevOps Example:

#!/usr/bin/env bash
# Automated log rotation and archiving

LOG_FILE="/var/log/app/application.log"
ARCHIVE_DIR="/var/log/archive"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create archive directory
mkdir -p "$ARCHIVE_DIR"

# Archive current log
cat "$LOG_FILE" | tee "$ARCHIVE_DIR/app_$TIMESTAMP.log"

# Truncate log file
> "$LOG_FILE"

echo "Log archived to $ARCHIVE_DIR/app_$TIMESTAMP.log"

---

File Information and Metadata

stat - Detailed File Information

# Basic stat output
stat /etc/hostname

# Custom format
stat -c '%n is %s bytes' /etc/hostname
stat -c 'Modified: %y' /etc/hostname
stat -c 'Access: %x, Modify: %y, Change: %z' /etc/hostname

# Stat multiple files
stat /etc/passwd /etc/hostname

# Check file type
stat -c '%A (%a) - %F' /etc/hostname

Format specifiers:

• %n - File name
• %s - Size in bytes
• %A - Permissions in human form
• %a - Permissions in octal
• %F - File type
• %x - Last access time
• %y - Last modification time
• %z - Last change time

file - Determine File Type

# Basic file type
file /etc/hostname
file /bin/ls
file /var/log/syslog

# Don't pause for "file" command
file -s /dev/sda1

# Show MIME type
file -i /etc/passwd

# Multiple files
file /etc/*

---

File Permissions

Understanding Permissions

Permission Structure: drwxr-xrwx
|  |  |  |  |  |  |
|  |  |  |  |  |  +-- Others: rwx (7)
|  |  |  |  |  +----- Group: rwx (7)
|  |  |  |  +---------- Owner: rwx (7)
|  |  |  +------------- Directory marker
+--+--+---------------- Type (d=dir, -=file, l=link)

Numeric Permissions

Number	Permission	Binary
0	---	000
1	—x	001
2	-w-	010
3	-wx	011
4	r—	100
5	r-x	101
6	rw-	110
7	rwx	111

Common Permission Sets

# 755 - Standard for executables and directories
# rwxr-xr-x

# 644 - Standard for regular files
# rw-r--r--

# 600 - Private files (owner only)
# rw-------

# 700 - Private directories
# rwx------

# 777 - World writable (AVOID!)
# rwxrwxrwx

chmod - Change Permissions

# Symbolic mode
chmod u+x script.sh          # Add execute for owner
chmod g-w file.txt           # Remove write for group
chmod o=rw file.txt          # Set others to read/write
chmod a+x script.sh          # Add execute for all
chmod +x script.sh           # Add execute for all (same)

# Numeric mode
chmod 755 script.sh          # rwxr-xr-x
chmod 644 config.txt         # rw-r--r--
chmod 600 .env               # rw-------
chmod 700 /secret/dir       # rwx------

# Recursive
chmod -R 755 /webapp        # All files in directory
chmod -R +X /webapp         # Only directories and add execute

chown - Change Ownership

# Change owner
chown user file.txt

# Change owner and group
chown user:group file.txt

# Change group only
chown :group file.txt

# Recursive
chown -R nginx:nginx /var/www

# Change owner of symbolic link (not target)
chown -h user link.txt

---

Searching for Files

find - The Power Tool

# Find by name
find /etc -name "*.conf"
find /var -name "*.log"

# Case insensitive
find /etc -iname "*.Conf"

# Find by type
find / -type f              # Regular files
find / -type d              # Directories
find / -type l              # Symbolic links
find / -type b              # Block devices

# Find by time
find /var/log -mtime -7     # Modified in last 7 days
find /var/log -mtime +30   # Modified more than 30 days ago
find /var/log -atime -1     # Accessed in last 24 hours
find /var/log -ctime -1    # Changed in last 24 hours

# Find by size
find / -size +100M          # Files larger than 100MB
find / -size -1K            # Files smaller than 1KB
find / -size 0              # Empty files

# Find by permissions
find / -perm 644            # Exactly 644
find / -perm -644           # At least 644 (includes 755)
find / -perm /u=x           # Owner executable

# Find and execute
find /var/log -name "*.log" -exec wc -l {} \;
find /etc -name "*.conf" -exec grep "Listen" {} \;

# Find and delete
find /tmp -type f -mtime +7 -delete

# Find and archive
find /var/www -name "*.php" -exec tar -czf php_files.tar.gz {} +

Real-world DevOps Examples:

# Find large log files
find /var/log -type f -size +100M

# Find configuration files
find /etc -name "*.conf" -o -name "*.cfg"

# Find files modified in last hour
find /var/www/html -type f -mmin -60

# Find executable scripts
find /opt -type f -perm -u+x

# Find and change ownership
find /var/www -type f -exec chown www-data:www-data {} \;

# Find files owned by specific user
find / -user nginx -type f

locate - Fast File Search

# Install mlocate first (sudo pacman -S mlocate)
sudo updatedb

# Quick search (uses database)
locate nginx.conf
locate -i readme          # Case insensitive

# Limit results
locate -n 20 "*.log"

# Only existing files
locate -e "*.conf"

which - Find Command Location

# Find command path
which python3
which kubectl
which docker

# Find all occurrences
which -a python3

whereis - Find Binary, Source, Manual

whereis python3
# python3: /usr/bin/python3 /usr/share/man/man1/python3.1.gz

---

File Comparison

diff - Compare Files

# Compare two files
diff file1.txt file2.txt

# Side-by-side comparison
diff -y file1.txt file2.txt

# Unified diff (git style)
diff -u file1.txt file2.txt

# Ignore case
diff -i file1.txt file2.txt

# Ignore whitespace
diff -w file1.txt file2.txt

# Recursive directory comparison
diff -r dir1/ dir2/

cmp - Binary Comparison

# Quick comparison
cmp file1.txt file2.txt

# Silent (exit code only)
cmp -s file1.txt file2.txt && echo "Identical"

comm - Line-by-Line Comparison

# Compare sorted files
comm file1.txt file2.txt

# Only lines unique to file1
comm -23 file1.txt file2.txt

# Only lines unique to file2
comm -13 file1.txt file2.txt

# Only common lines
comm -12 file1.txt file2.txt

---

File Compression

tar - Archive Files

# Create archive
tar -cf archive.tar file1 file2 dir1/
tar -cvf archive.tar file1 file2   # Verbose

# Extract archive
tar -xf archive.tar
tar -xvf archive.tar              # Verbose

# List contents
tar -tf archive.tar

# Create gzipped archive (most common)
tar -czvf archive.tar.gz dir1/
tar -czf archive.tar.gz dir1/

# Extract gzipped
tar -xzvf archive.tar.gz
tar -xzf archive.tar.gz

# Create bz2 archive
tar -cjvf archive.tar.bz2 dir1/

# Create xz archive (best compression)
tar -cJvf archive.tar.xz dir1/

# Extract to specific directory
tar -xf archive.tar -C /destination/

# Extract specific file
tar -xf archive.tar specific_file.txt

# Add to existing archive
tar -rf archive.tar newfile.txt

# Update existing archive
tar -uf archive.tar modified_file.txt

gzip / gunzip

# Compress file
gzip file.txt
# Results in file.txt.gz

# Keep original
gzip -k file.txt

# Compress to stdout
gzip -c file.txt > file.txt.gz

# Decompress
gunzip file.txt.gz
gzip -d file.txt.gz

# View without extracting
zcat file.txt.gz
zless file.txt.gz
zgrep "pattern" file.txt.gz

zip / unzip

# Create zip archive
zip archive.zip file1 file2

# Recursive
zip -r archive.zip directory/

# Exclude files
zip -r archive.zip directory/ -x "*.git/*"

# Extract
unzip archive.zip

# Extract to directory
unzip archive.zip -d /destination/

# List contents
unzip -l archive.zip

# Password protect
zip -P password archive.zip file.txt

---

File Editing

sed for In-Place Editing

# Replace text in file (macOS requires empty string for backup)
sed -i 's/old/new/g' file.txt

# Create backup before editing
sed -i.bak 's/old/new/g' file.txt

# Delete lines matching pattern
sed -i '/pattern/d' file.txt

# Insert line before pattern
sed -i '/pattern/i\New line' file.txt

# Append line after pattern
sed -i '/pattern/a\New line' file.txt

Using vim/nano for Interactive Editing

# Open with vim
vim file.txt
nano file.txt

# Non-interactive editing
ex -s +'%s/old/new/g' +wq file.txt

---

Working with Directories

mkdir - Create Directories

# Basic
mkdir newdir

# Create parent directories
mkdir -p /path/to/nested/dir

# Create with permissions
mkdir -m 755 newdir

# Verbose
mkdir -pv newdir

rmdir - Remove Empty Directories

# Basic
rmdir emptydir

# Remove parent if empty
rmdir -p parent/child

rm - Remove Files and Directories

# Remove file
rm file.txt

# Remove directory and contents
rm -rf directory/

# Interactive
rm -i file.txt

# Verbose
rm -rv directory/

# Force (no prompts)
rm -f file.txt

# Preserve root (safety)
rm -rf /  # DON'T DO THIS!

---

Special File Operations

touch - Create Empty File / Update Timestamp

# Create empty file
touch newfile.txt

# Update timestamp
touch existingfile.txt

# Create with specific timestamp
touch -d "2024-01-01 00:00:00" file.txt
touch -t 202401010000 file.txt

# Create multiple files
touch file{1..10}.txt

ln - Create Links

# Hard link
ln source.txt hardlink.txt

# Symbolic (soft) link
ln -s source.txt symlink.txt

# Symbolic link to directory
ln -s /var/www/html website

# Update link
ln -sf newtarget symlink

cp - Copy Files

# Copy file
cp source.txt destination.txt

# Copy to directory
cp file.txt /destination/

# Preserve attributes
cp -p file.txt /destination/

# Recursive
cp -r directory/ /destination/

# Interactive (ask before overwrite)
cp -i file.txt /destination/

# Update (only if newer)
cp -u file.txt /destination/

# Archive (preserves everything)
cp -a source/ destination/

mv - Move/Rename Files

# Move file
mv source.txt /destination/

# Rename file
mv oldname.txt newname.txt

# Move and rename
mv source.txt /destination/newname.txt

# Interactive
mv -i source.txt /destination/

# Force (no prompts)
mv -f source.txt /destination/

---

Summary

In this chapter, you learned:

• ✅ Reading files with cat, less, head, tail
• ✅ Writing and appending to files
• ✅ Using tee for logging
• ✅ Understanding file metadata with stat
• ✅ File permissions (chmod, chown)
• ✅ Finding files (find, locate, which)
• ✅ File comparison (diff, cmp, comm)
• ✅ File compression (tar, gzip, zip)
• ✅ File editing (sed, vim)
• ✅ Directory operations
• ✅ Links, copy, move operations

---

Next Steps

Continue to the next chapter to learn about Here Documents and Here Strings.

---

Previous Chapter: Arguments Next Chapter: Here Documents