Sed

Chapter 21: sed - Stream Editor

Overview

sed (Stream Editor) is a powerful text processing utility that performs editing operations on streams of text. For DevOps engineers, sed is essential for config file modifications, log processing, and automated deployments.

---

How sed Works

┌────────────────────────────────────────────────────────────────┐
│                       sed Processing Flow                      │
├────────────────────────────────────────────────────────────────┤
│                                                                │
│   Input File          sed Engine           Output              │
│   ─────────          ───────────           ──────              │
│                                                                │
│   line 1     ───►  ┌─────────────┐     ──►   line 1           │
│   line 2     ───►  │ Read line   │     ──►   (modified)      │
│   line 3     ───►  │ Apply cmds  │     ──►   line 2           │
│   ...       ───►  │ Output      │     ──►   (modified)      │
│                   └─────────────┘     ──►   line 3           │
│                                              ...               │
│                                                                │
│   Cycle: Read → Pattern Space → Commands → Print → Repeat    │
│                                                                │
└────────────────────────────────────────────────────────────────┘

---

Basic Syntax

# Basic syntax
sed [options] 'command' input_file

# Common options
-n              # Suppress automatic output
-e              # Multiple commands
-i              # In-place editing
-r              # Extended regex
-f              # Read commands from file

---

Substitution Command (s)

Basic Substitution

# Replace first occurrence in each line
sed 's/old/new/' file.txt

# Replace all occurrences (global)
sed 's/old/new/g' file.txt

# Replace on specific line
sed '3s/old/new/' file.txt

# Replace in range of lines
sed '1,5s/old/new/' file.txt

Case Insensitive Replace

# Case insensitive
sed 's/old/new/gi' file.txt

# Example
echo "Hello WORLD" | sed 's/world/Universe/gi'
# Hello Universe

Delimiter Options

# Use different delimiter (useful for paths)
sed 's|/etc/nginx|/opt/nginx|g' file.conf

# Using # as delimiter
sed 's#http://#https://#g' file.html

# Using | as delimiter
sed 's|old|new|g' file.txt

---

Address Ranges

Line Numbers

# Specific line
sed '5s/old/new/' file.txt

# Range of lines
sed '5,10s/old/new/' file.txt

# From line to end
sed '5,$s/old/new/' file.txt

# First few lines
sed '1,3s/old/new/' file.txt

Pattern Matching

# Lines matching pattern
sed '/error/s/old/new/' file.txt

# Range from pattern to pattern
sed '/start/,/end/s/old/new/' file.txt

# Multiple patterns
sed -e '/pattern1/s/a/b/' -e '/pattern2/s/c/d/' file.txt

---

Deletion Commands

Delete Lines

# Delete specific line
sed '5d' file.txt

# Delete range
sed '5,10d' file.txt

# Delete from line to end
sed '5,$d' file.txt

# Delete matching lines
sed '/error/d' file.txt

# Delete empty lines
sed '/^$/d' file.txt

# Delete lines starting with #
sed '/^#/d' file.txt

# Delete lines starting with whitespace and #
sed '/^[[:space:]]*#/d' file.txt

---

Insert and Append

Insert Before/After

# Insert before line 5
sed '5i\New line content' file.txt

# Insert before first match of pattern
sed '/pattern/i\New line before' file.txt

# Append after line 5
sed '5a\New line after' file.txt

# Append after first match of pattern
sed '/pattern/a\New line after' file.txt

Practical Example

# Add header to file
sed '1i\=== Header Line ===' file.txt

# Add footer to file
sed '$a\=== Footer Line ===' file.txt

---

Transform Command (y)

Character Translation

# Transform characters (like tr)
sed 'y/abc/123/' file.txt
# a→1, b→2, c→3

# Convert to uppercase
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' file.txt

# Convert to lowercase (GNU sed)
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' file.txt

---

Holding Pattern

Using Hold Space

# h - Copy pattern to hold
# H - Append pattern to hold
# g - Copy hold to pattern
# G - Append hold to pattern
# x - Exchange pattern and hold

# Reverse lines
sed '1!G;h;$!d' file.txt

# Extract first and last line
sed -n '1p;$p' file.txt

---

Multiple Commands

Using -e

# Multiple edits
sed -e 's/old1/new1/' -e 's/old2/new2/' file.txt

# Chain of operations
sed -e '1i\Header' -e '/error/d' -e 's/warn/INFO/g' file.txt

Using semicolon

# Multiple commands on same line
sed 's/a/b/;s/c/d/;s/e/f/' file.txt

# With addresses
sed '1,5s/old/new/;10d' file.txt

---

Regular Expressions in sed

Basic vs Extended

# Basic regex (default)
sed 's/[0-9]*/NUM/g' file.txt

# Extended regex (-r flag)
sed -r 's/[0-9]+/NUM/g' file.txt

# Using | for alternation (requires -r)
sed -r 's/(error|warn|info)/LEVEL/g' file.txt

Pattern Examples

# Match IP address
sed -E 's/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/IP_ADDR/g' file.txt

# Match email
sed -E 's/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/EMAIL/g' file.txt

# Match timestamp
sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2}/YYYY-MM-DD/g' file.txt

---

Practical DevOps Examples

1. Replace Configuration Values

# Replace port in nginx config
sed -i 's/listen 80/listen 8080/' /etc/nginx/nginx.conf

# Replace database host
sed -i 's/db_host=localhost/db_host=postgres/' .env

# Replace multiple values
sed -i -e 's/DEBUG=true/DEBUG=false/' \
       -e 's/LOG_LEVEL=debug/LOG_LEVEL=info/' \
       -e 's/MAX_WORKERS=4/MAX_WORKERS=8/' config.yaml

2. Modify Log Files

# Remove ANSI color codes
sed -r 's/\x1B\[[0-9;]*[JKmsu]//g' logfile

# Remove timestamps
sed -E 's/[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}//g' logfile

# Extract specific content
sed -n '/ERROR/p' application.log

3. Docker and ContainerOps

# Remove stopped containers
docker ps -a | grep Exit | sed 's/ .*//' | xargs docker rm

# Remove dangling images
docker images | grep '<none>' | awk '{ print $3 }' | xargs docker rmi

# Extract container IP
docker inspect container_name | sed -n '/IPAddress/{s/.*: //;s/",//;p}'

4. Process Kubernetes Logs

# Extract specific fields from JSON logs
kubectl logs deployment/app | sed 's/{"time":"\([^"]*\)".*/\1/'

# Remove debug entries
kubectl logs app | sed '/DEBUG/d'

# Replace sensitive data
kubectl logs app | sed -E 's/password=[^&]*/password=***/g'

5. AWS CLI Output Processing

# Parse EC2 instance IDs
aws ec2 describe-instances | sed -n '/InstanceId/{s/.*": "//;s/",//;p}'

# Extract security group IDs
aws ec2 describe-security-groups | sed -n '/GroupId/{s/.*": "//;s/",//;p}'

# Parse S3 ls output
aws s3 ls s3://bucket/ | sed 's/.* \([0-9-]*\) .*/\1/'

6. Git Operations

# Replace remote URL
git remote set-url origin $(git remote get-url origin | sed 's/https:\/\/github.com/git@github.com:/')

# Clean up branches
git branch | sed 's/^* //' | grep 'feature-' | xargs git branch -d

# Update commit messages
git filter-branch --msg-filter 'sed "s/old/new/"'

7. File Cleanup

# Remove comments
sed '/^[[:space:]]*#/d' config.txt

# Remove blank lines
sed '/^$/d' file.txt

# Remove Windows line endings
sed -i 's/\r$//' file.txt

# Add line numbers
sed = file.txt | sed 'N;s/\n/\t/'

8. System Administration

# Update /etc/hosts
echo "192.168.1.100 server1" | sed -i '$a\' /etc/hosts

# Modify fstab (careful!)
sed -i 's/ext3/ext4/' /etc/fstab

# Update sysctl values
sed -i 's/net.ipv4.ip_forward = 0/net.ipv4.ip_forward = 1/' /etc/sysctl.conf

---

In-Place Editing

Safe Editing

# Edit in place (creates backup)
sed -i.bak 's/old/new/g' file.txt
# Creates file.txt.bak as backup

# Edit in place (no backup)
sed -i 's/old/new/g' file.txt

# Edit multiple files
sed -i 's/old/new/g' file1.txt file2.txt file3.txt

Using symlink

# Edit through symlink (preserves symlink)
sed -i --follow-symlinks 's/old/new/g' /path/to/symlink

---

Common sed Recipes

Text Replacement

# Replace first occurrence only
sed 's/old/new/'

# Replace last occurrence
sed 's/.*old/new/'

# Replace word only
sed 's/\bold\b/new/g'

# Replace between delimiters
sed '/start/,/end/s/old/new/g'

Line Operations

# Show specific lines
sed -n '5p' file.txt
sed -n '5,10p' file.txt
sed -n '5~2p' file.txt  # Every 2nd line from 5

# Duplicate lines
sed 'p' file.txt

# Duplicate each line
sed 'p;p' file.txt

# Print line numbers
sed '=' file.txt

Whitespace Operations

# Remove leading whitespace
sed 's/^[[:space:]]*//' file.txt

# Remove trailing whitespace
sed 's/[[:space:]]*$//' file.txt

# Remove both
sed 's/^[[:space:]]*//;s/[[:space:]]*$//' file.txt

# Convert tabs to spaces
sed 's/\t/    /g' file.txt

# Compress multiple spaces
sed 's/[[:space:]]\+/ /g' file.txt

---

Summary

In this chapter, you learned:

• ✅ How sed works (pattern space, hold space)
• ✅ Basic substitution commands
• ✅ Address ranges (lines and patterns)
• ✅ Insert, append, and delete operations
• ✅ Character transformation with y command
• ✅ Regular expressions in sed
• ✅ Practical DevOps examples
• ✅ In-place editing and safety

---

Next Steps

Continue to the next chapter to learn about awk - the powerful pattern scanning and processing language.

---

Previous Chapter: Regular Expressions Next Chapter: awk - Pattern Scanning