Logging

Chapter 52: Logging and Log Management

Overview

Logging is critical for system administration, security auditing, troubleshooting, and compliance. Linux provides robust logging infrastructure including rsyslog, journald, and various log management tools. This chapter covers everything needed for effective log management.

52.1 Linux Logging Architecture

Logging Components

                     Linux Logging Architecture
+------------------------------------------------------------------+
|                                                                   |
|                    +------------------------+                     |
|                    |  Logging Components    |                     |
|                    +------------------------+                     |
|                              |                                    |
|        +---------------------+---------------------+              |
|        |                     |                     |              |
|        v                     v                     v              |
|  +----------+          +----------+          +----------+        |
|  |Applications|        |rsyslog   |          |journald  |        |
|  |           |        |          |          |          |        |
|  | - Services|        | - Local  |          | - systemd|        |
|  | - Daemons |        |   logs   |          |   logs  |        |
|  +----------+          +----------+          +----------+        |
|        |                     |                     |              |
|        +---------------------+---------------------+              |
|                              |                                    |
|        +---------------------+---------------------+              |
|        |                     |                     |              |
|        v                     v                     v              |
|  +----------+          +----------+          +----------+        |
|  |/var/log/ |          |/var/log/ |          | Journal  |        |
|  |messages  |          | syslog   |          |Database  |        |
|  +----------+          +----------+          +----------+        |
|                                                                   |
+------------------------------------------------------------------+

B --> J
J --> L2
R --> L1

### Key Log Files

```bash
# System logs
/var/log/messages            # General system messages (Debian/Ubuntu)
/var/log/syslog              # General system messages (Arch/RHEL)
/var/log/dmesg               # Kernel ring buffer
/var/log/kern.log            # Kernel messages
/var/log/boot.log            # Boot messages

# Application logs
/var/log/nginx/
    access.log               # HTTP access logs
    error.log                # Error logs
/var/log/apache2/
    access.log
    error.log
/var/log/postgresql/
    postgresql.log
/var/log/mysql/
    error.log
    slow.log

# Security logs
/var/log/auth.log            # Authentication logs (Debian)
/var/log/secure              # Authentication logs (RHEL/Arch)
/var/log/faillog             # Failed login attempts
/var/log/lastlog             # Last login records

# Systemd journal
journalctl                   # Query systemd journal

52.2 Systemd Journal (journald)

Basic journalctl Usage

# View all logs (paginated)
journalctl

# View kernel messages
journalctl -k
journalctl --dmesg

# View logs for specific service
journalctl -u nginx.service
journalctl -u postgresql

# View logs since specific time
journalctl --since "2024-01-01"
journalctl --since "1 hour ago"
journalctl --since "yesterday"

# View logs in time range
journalctl --since "2024-01-01" --until "2024-01-02"

# Follow logs in real-time
journalctl -f
journalctl -fu nginx.service

# View recent logs
journalctl -n 100
journalctl -n 100 --no-pager

# Filter by priority
# 0=emergency, 1=alert, 2=critical, 3=error, 4=warning, 5=notice, 6=info, 7=debug
journalctl -p err
journalctl -p warning..emerg

Advanced Filtering

# By process ID
journalctl _PID=1234

# By user ID
journalctl _UID=1000

# By executable
journalctl /usr/bin/nginx

# By device
journalctl _DEVICE=/dev/sda1

# By systemd unit
journalctl -u nginx.service -u php-fpm.service

# Combined filters
journalctl -u nginx --since "1 hour ago" -p error

# Boot-specific logs
journalctl --list-boots
journalctl -b -1              # Previous boot
journalctl -b 2              # Boot 2 ago

Journal Configuration

# View current configuration
cat /etc/systemd/journald.conf

# Configuration options
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M            # Max disk usage
RuntimeMaxUse=100M           # Max in-memory storage
MaxRetentionSec=1month       # Max retention time
MaxFileSec=1week             # Rotate weekly
Storage=persistent           # persistent, volatile, auto, none
Compress=yes                 # Compress old entries
ForwardToSyslog=yes          # Forward to rsyslog

Journal Storage Management

# Disk usage
journalctl --disk-usage

# Vacuum old logs
journalctl --vacuum-size=100M      # Keep only 100MB
journalctl --vacuum-time=2weeks    # Keep 2 weeks
journalctl --vacuum-files=5        # Keep 5 files

# Make journal persistent
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

52.3 rsyslog

rsyslog Configuration

# Main configuration file
/etc/rsyslog.conf

# Modules
$ModLoad imuxsock            # Local message input
$ModLoad imklog              # Kernel messages
$ModLoad imudp               # UDP receiver
$ModLoad imtcp               # TCP receiver

# Rules format
# facility.priority   action

# Facilities: auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, local0-7
# Priorities: debug, info, notice, warning, err, crit, alert, emerg

Rule Examples

# Log everything to file
*.*                         /var/log/all.log

# Log auth messages
auth.*                      /var/log/auth.log
authpriv.*                  /var/log/secure

# Log mail
mail.*                      /var/log/maillog
mail.err                    /var/log/mail.err

# Log cron
cron.*                      /var/log/cron

# Log kernel
kern.*                      /var/log/kern.log

# Discard (don't log)
mail.*                      ~

# Forward to remote server
*.*                         @remote-host:514
*.*                         @@remote-host:514    # TCP

# Log to console
*.emerg                     /dev/console

Remote Logging

# Server configuration (/etc/rsyslog.conf)
# Enable UDP/TCP reception
module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Client configuration
*.* @rsyslog-server.example.com:514

# TLS encrypted logging
$DefaultNetstreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedNode rsyslog-client
*.* @@(o)gsecs.tls.example.com:6514

52.4 Log Rotation

logrotate Configuration

# Main configuration
/etc/logrotate.conf

# Include directory for service-specific configs
include /etc/logrotate.d/

# Example logrotate.conf
daily                       # Rotate daily
rotate 14                   # Keep 14 days
compress                    # Compress old logs
delaycompress               # Don't compress immediately
missingok                   # OK if file missing
notifempty                  # Don't rotate if empty
create 0640 root adm        # Permissions for new logs
dateext                     # Date-based rotation
dateformat -%Y%m%d          # Date format

# Additional options
maxsize 100M                # Rotate if > 100MB
minsize 10M                 # Don't rotate if < 10MB
size 100M                   # Size-based rotation
sharedscripts               # Run postrotate once per rotation
prerotate/endscript         # Pre-rotation commands
postrotate/endscript        # Post-rotation commands

Service-Specific Configuration

/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

# /etc/logrotate.d/apache2
/var/log/apache2/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if [ -f /var/run/apache2/apache2.pid ]; then
            /usr/sbin/invoke-rc.d apache2 graceful > /dev/null
        fi
    endscript
}

# /etc/logrotate.d/syslog
/var/log/cron.log
/var/log/maillog
/var/log/messages
/var/log/syslog {
    daily
    rotate 7
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Manual Log Rotation

# Force rotation
logrotate -f /etc/logrotate.conf
logrotate -d -f /etc/logrotate.d/nginx  # Debug mode

# Rotate all logs now
logrotate -f /etc/logrotate.conf

52.5 Analyzing Logs

Common Log Analysis Tasks

# Count lines in log
wc -l /var/log/messages

# Find errors
grep -i error /var/log/messages
grep -E "error|fail|critical" /var/log/messages

# Find specific time window
sed -n '/Jan 15 10:00/,/Jan 15 11:00/p' /var/log/messages

# Failed SSH attempts
grep "Failed password" /var/log/auth.log
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

# Most common errors
grep error /var/log/messages | awk '{print $5}' | sort | uniq -c | sort -rn

# Disk space issues
grep -i "no space" /var/log/messages

# Service restarts
grep -i "started\|stopped\|restarted" /var/log/messages | tail -20

Log Analysis Commands

# tail -f for live monitoring
tail -f /var/log/messages
tail -f /var/log/nginx/error.log

# Watch command
watch -n 1 'tail -20 /var/log/messages'

# awk examples
awk '/error/ {print $1,$2,$5}' /var/log/messages
awk '/ssh/ {print $1,$2,$3,$11}' /var/log/auth.log

# sed for extraction
sed -n '/Failed password/,/192.168/p' /var/log/auth.log

# cut for column extraction
cut -d' ' -f1-4 /var/log/messages | head

Logwatch

# Install logwatch
sudo pacman -S logwatch

# Configuration
/etc/logwatch/conf/logwatch.conf

# Run manually
/usr/sbin/logwatch --output stdout --detail high --service all

# Email report
/usr/sbin/logwatch --output mail --mailto admin@example.com --detail high

52.6 Centralized Logging

ELK Stack Overview

                          ELK Stack Architecture
+------------------------------------------------------------------+
|                                                                   |
|  Clients                Logstash      Elasticsearch    Grafana    |
|  +--------+            +--------+      +-----------+   +------+  |
|  |Server 1|───────────>|        |      |           |   |      |  |
|  +--------+            |        |      |           |   |      |  |
|  +--------+   rsyslog | Logstash|------>|Elasticsearch|-->  |Grafana| |
|  |Server 2|───────────>|        |      |           |   |      |  |
|  +--------+            |        |      |           |   |      |  |
|  +--------+            +--------+      +-----------+   +------+  |
|  |Server N|                                                         |
|  +--------+                                                         |
|                                                                   |
+------------------------------------------------------------------+

Setting Up Filebeat

# Install filebeat
sudo pacman -S filebeat

# Configuration
# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log
      - /var/log/nginx/*.log
    fields:
      service: syslog
    fields_under_root: true

output.logstash:
  hosts: ["logstash.example.com:5044"]

# Enable filebeat modules
filebeat modules enable system
filebeat modules enable nginx

# Start service
sudo systemctl enable --now filebeat

Loki (Prometheus Stack)

# Docker-compose for Loki
# docker-compose.yml
version: '3'
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./config:/etc/loki
      - ./data:/loki

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./config:/etc/promtail
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

52.7 Log Security

Protecting Log Files

# Set proper permissions
chmod 640 /var/log/messages
chmod 640 /var/log/secure
chown root:adm /var/log/messages

# Immutable logs (append only)
chattr +a /var/log/messages
# To remove: chattr -a /var/log/messages

# Audit logging for log access
auditctl -w /var/log/messages -p wa -k log_modifications

# Monitor log changes
inotifywait -m /var/log/ -e modify -e create -e delete

Compliance Considerations

# Ensure logs are not tampered with
# Use write-once media for critical logs
# Implement log signing for integrity
# Regular backup of logs

# PCI-DSS requirements
# - Log retention for 1 year
# - 3 months easily accessible
# - File integrity monitoring
# - Time synchronization

52.8 Troubleshooting Examples

High Disk Space from Logs

# Find largest log files
du -sh /var/log/*
du -sh /var/log/* | sort -rh | head -10

# Find largest directories
du -shx /var/log/* 2>/dev/null | sort -rh | head -10

# Check for log explosion
ls -lahS /var/log | head -10

Service Not Logging

# Check if service logs to journal
journalctl -u service-name -n 50

# Check rsyslog running
systemctl status rsyslog

# Check journald running
systemctl status systemd-journald

# Check log directory permissions
ls -la /var/log/service/

# Restart logging services
sudo systemctl restart rsyslog
sudo systemctl restart systemd-journald

Application Not Logging

# Check application configuration
# Ensure logging level is not set to 'quiet'
# Check log file path exists

# For systemd services
# Add StandardOutput=file:/var/log/app.log
# Add StandardError=file:/var/log/app-error.log

52.9 Practical Scripts

Log Analysis Script

#!/bin/bash
LOGFILE=${1:-/var/log/messages}
REPORT="/tmp/log_report_$(date +%Y%m%d).txt"

echo "=== Log Analysis Report ===" > "$REPORT"
echo "Generated: $(date)" >> "$REPORT"
echo "" >> "$REPORT"

# Error count
echo "=== Error Summary ===" >> "$REPORT"
grep -ci error "$LOGFILE" >> "$REPORT" 2>/dev/null || echo "0" >> "$REPORT"

# Top 10 errors
echo "" >> "$REPORT"
echo "=== Top 10 Errors ===" >> "$REPORT"
grep -i error "$LOGFILE" 2>/dev/null | awk '{print $5}' | sort | uniq -c | sort -rn | head -10 >> "$REPORT"

# Failed logins
echo "" >> "$REPORT"
echo "=== Failed Login Attempts ===" >> "$REPORT"
grep "Failed password" "$LOGFILE" 2>/dev/null | tail -20 >> "$REPORT"

# Disk space warnings
echo "" >> "$REPORT"
echo "=== Disk Space Warnings ===" >> "$REPORT"
grep -i "no space\|disk full\|quota" "$LOGFILE" 2>/dev/null | tail -10 >> "$REPORT"

# Service restarts
echo "" >> "$REPORT"
echo "=== Service Restarts ===" >> "$REPORT"
grep -i "started\|stopped" "$LOGFILE" 2>/dev/null | tail -20 >> "$REPORT"

cat "$REPORT"

Summary

In this chapter, you learned:

✅ Linux logging architecture (rsyslog, journald)
✅ Using journalctl for systemd logs
✅ rsyslog configuration and rules
✅ Log rotation with logrotate
✅ Log analysis techniques
✅ Centralized logging with ELK/Loki
✅ Log security and compliance
✅ Troubleshooting log issues

Next Chapter

Chapter 10: Monitoring and Alerting

Last Updated: February 2026