Logging

Chapter 36: System Logging - rsyslog, syslog-ng Deep Dive

Mastering Linux System Logging for Production Environments

36.1 Understanding Linux Logging Architecture

The Linux Logging Ecosystem

Linux logging is a multi-layered system that captures, processes, and stores system and application events. Understanding this architecture is crucial for effective system administration and troubleshooting.

                     Linux Logging Architecture
+------------------------------------------------------------------+
|                                                                   |
|                        Application Layer                             |
|    +-------------+ +-------------+ +-------------+                 |
|    |  nginx     | |  sshd     | |  systemd   |                 |
|    +-------------+ +-------------+ +-------------+                 |
|           |               |               |                        |
|           v               v               v                        |
|    +-------------------------------------------------------------+ |
|    |              syslog() / journalctl API                        | |
|    +-------------------------------------------------------------+ |
|                              |                                      |
|                              v                                      |
|                        Kernel Layer                                |
|    +-------------------------------------------------------------+ |
|    |              /proc/kmsg (Kernel Messages)                     | |
|    +-------------------------------------------------------------+ |
|                              |                                      |
|                              v                                      |
|                    System Logging Daemons                          |
|    +-------------+ +-------------+ +-------------+                 |
|    |  rsyslogd  | | syslog-ng | | systemd-journald|            |
|    +-------------+ +-------------+ +-------------+                 |
|                              |                                      |
|              +---------------+---------------+                      |
|              |               |               |                      |
|              v               v               v                      |
|        Local Files      Remote syslog   Database                  |
|    /var/log/messages    Central server   (MySQL/PostgreSQL)       |
|                                                                   |
+------------------------------------------------------------------+

Syslog Facilities and Priorities

                     Syslog Facilities (0-23)
+------------------------------------------------------------------+
|                                                                   |
|    0 (kern)     - Kernel messages                                |
|    1 (user)     - User-level messages                           |
|    2 (mail)     - Mail system                                   |
|    3 (daemon)   - System daemons                                |
|    4 (auth)     - Authentication messages                       |
|    5 (syslog)   - Syslogd internal messages                    |
|    6 (lpr)      - Line printer subsystem                        |
|    7 (news)     - Network news subsystem                        |
|    8 (uucp)     - UUCP subsystem                               |
|    9 (cron)     - Clock daemon                                  |
|   10 (authpriv) - Private authentication                        |
|   11 (ftp)      - FTP daemon                                    |
| 12-23          - Local use 0-11                                |
|                                                                   |
|    Syslog Priorities (0-7)                                       |
|    +----------------------------------------------------------+   |
|    | 0 (emerg)   - Emergency: system is unusable             |   |
|    | 1 (alert)  - Alert: must be taken immediately          |   |
|    | 2 (crit)   - Critical: critical conditions             |   |
|    | 3 (err)    - Error: error conditions                  |   |
|    | 4 (warn)   - Warning: warning conditions              |   |
|    | 5 (notice) - Notice: normal but significant            |   |
|    | 6 (info)   - Informational: informational             |   |
|    | 7 (debug)  - Debug: debug-level messages               |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

36.2 rsyslog Deep Dive

rsyslog Architecture

rsyslog is the most widely used syslog daemon on Linux, offering reliable log forwarding, filtering, and output formatting.

                     rsyslog Processing Pipeline
+------------------------------------------------------------------+
|                                                                   |
|    Input Module        Processing           Output Module           |
|    +----------+       +---------+         +----------+            |
|    | imuxsock |<---->| Queue   |<------>| omfile   |            |
|    +----------+       +---------+         +----------+            |
|    | imtcp    |       | Filter  |         | omfwd    |            |
|    +----------+       +---------+         +----------+            |
|    | imudp    |       | Template|         | omprog   |            |
|    +----------+       +---------+         +----------+            |
|    | imjournal|       | Parser  |         | ommysql  |            |
|    +----------+       +---------+         +----------+            |
|                                                                   |
|    Key Concepts:                                                   |
|    +----------------------------------------------------------+   |
|    |  - Input: Where logs come from                           |   |
|    |  - Queue: Buffer before processing                       |   |
|    |  - Filter: Select/drop messages based on rules          |   |
|    |  - Template: Format output                                |   |
|    |  - Output: Where logs go                                |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

Comprehensive rsyslog Configuration

# /etc/rsyslog.conf - Production Configuration

# =============================================================================
# MODULES
# =============================================================================

# Load modules
module(load="imuxsock")      # Local logging via /dev/log
module(load="imjournal")      # Systemd journal
module(load="imklog")        # Kernel logging
module(load="imudp")          # UDP syslog reception
module(load="imtcp")          # TCP syslog reception
module(load="imrelp")         # RELP protocol (reliable)
module(load="ommysql")        # MySQL output
module(load="ompgsql")       # PostgreSQL output
module(load="omelasticsearch")# Elasticsearch output

# =============================================================================
# GLOBAL DIRECTIVES
# =============================================================================

# Default permissions for new log files
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Template for standard format
$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%"
$template RFC5424fmt,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32% %msg:::sp-if-no-msg%"

# JSON template for structured logging
$template json-log,"{\"time\":\"%TIMESTAMP%\",\"host\":\"%HOSTNAME%\",\"syslogtag\":\"%syslogtag%\",\"pri\":\"%PRI%\",\"msg\":\"%msg:::jsonf:::replace-all-cntrl:::jsonf-replace:::jsonf-replace%\"}%newline%"

# =============================================================================
# LOG FILTERS
# =============================================================================

# Kernel messages
kern.*                                                 /var/log/kern.log

# Critical and higher to console
*.crit                                                /dev/console

# Authentication
auth,authpriv.*                                       /var/log/secure

# Mail
mail.*                                                -/var/log/maillog
mail.*                                                @remote-logserver

# Cron
cron.*                                                /var/log/cron

# Emergency messages to all users
*.emerg                                               :omusrmsg:*

# All info but exclude mail and authpriv
*.info;mail.none;authpriv.none;cron.none              /var/log/messages

# Warnings to separate file
*.warn;*.err                                          /var/log/warnings

# Debug messages
*.=debug                                              /var/log/debug

# =============================================================================
# TEMPLATES
# =============================================================================

# Remote logging template with hostname
$template RemoteLogs,"/var/log/%HOSTNAME%/%programname%.log"
*.* ?RemoteLogs

# High-precision timestamps
$template precise,"%TIMESTAMP% %hostname% %syslogtag%%msg%"
*.* -?precise

# =============================================================================
# FORWARDING TO REMOTE SERVER
# =============================================================================

# UDP forwarding (default, faster but unreliable)
*.* @remote-server.example.com:514

# TCP forwarding (reliable)
*.* @@remote-server.example.com:514

# RELP forwarding (most reliable)
*.* :omrelp:remote-server.example.com:1514

# Forward with specific template
*.* @@remote-server.example.com:514;RFC3164fmt

# =============================================================================
# CONDITIONAL LOGGING
# =============================================================================

# Log to file based on program name
if $programname == 'nginx' then /var/log/nginx.log
if $programname == 'nginx' then stop

if $msg contains "error" then /var/log/errors.log
& stop

# Log specific severity
if $syslogseverity <= 3 then /var/log/critical.log
& stop

rsyslog Filtering Rules

# Drop messages from specific host
:hostname , !isequal , "trusted-host.example.com" ~

# Drop messages containing specific pattern
:msg , !contains , "heartbeat" ~

# Drop messages from specific facility
:facility, !isequal , "local7" ~

# Rate limiting (drop if more than 100 messages in 5 seconds)
:msg, startswith, "audit" rate-limit:100 0

# Discard after logging
*.* /var/log/everything.log
& ~

# Parse structured data
$syslogtag, contains, "apache2" then {
    action(type="omfile" file="/var/log/apache.log")
    action(type="omfwd" protocol="tcp" target="logs.example.com" port="514")
}

36.3 systemd Journal Deep Dive

Journal Architecture

                     systemd Journal Architecture
+------------------------------------------------------------------+
|                                                                   |
|    +-------------+     +-------------+     +-------------+        |
|    | Kernel      |     | Services    |     | Users       |        |
|    | /dev/kmsg   |     | journalctl  |     | systemd-sys |        |
|    +-------------+     +-------------+     +-------------+        |
|           |               |               |                        |
|           v               v               v                        |
|    +-------------------------------------------------------------+ |
|    |              journald (Journal Daemon)                       | |
|    |                                                             | |
|    |  - Collects logs from all sources                          | |
|    |  - Structured logging (key-value pairs)                     | |
|    |  - Indexing for fast searching                             | |
|    |  - Binary format                                           | |
|    +-------------------------------------------------------------+ |
|                    |                    |                          |
|                    v                    v                          |
|         /run/log/journal    /var/log/journal                     |
|         (volatile)           (persistent - if configured)          |
|                                                                   |
+------------------------------------------------------------------+

Comprehensive journalctl Usage

# =============================================================================
# BASIC VIEWING
# =============================================================================

# View all logs
journalctl

# View since last boot
journalctl -b
journalctl -b -1  # Previous boot

# View specific boot
journalctl -b abc123

# =============================================================================
# TIME-BASED FILTERING
# =============================================================================

# Since specific time
journalctl --since "2024-01-01 00:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday
journalctl --since "2024-01-01" --until "2024-01-02"

# Since specific date
journalctl --since "2024-01-01"

# Time range
journalctl --since "2024-01-01 10:00:00" --until "2024-01-01 11:00:00"

# =============================================================================
# PRIORITY FILTERING
# =============================================================================

# Priority levels: emerg(0), alert(1), crit(2), err(3), warning(4), notice(5), info(6), debug(7)
journalctl -p err          # Error and above
journalctl -p warning     # Warning and above
journalctl -p 0..3        # Emergency to error

# =============================================================================
# UNIT/SERVICE FILTERING
# =============================================================================

# Specific service
journalctl -u nginx.service
journalctl -u nginx.service -f  # Follow

# Multiple services
journalctl -u nginx.service -u mysql.service

# Failed services
journalctl --failed
journalctl -p 3 -b  # Errors since boot

# Services that crashed
journalctl --fault

# =============================================================================
# PROCESS FILTERING
# =============================================================================

# Specific PID
journalctl _PID=1234

# Specific user
journalctl _UID=1000
journalctl _UID=$(id -u username)

# Specific group
journalctl _GID=1000

# Executable path
journalctl _EXE=/usr/bin/nginx

# =============================================================================
# MESSAGE FILTERING
# =============================================================================

# Filter by message content
journalctl MESSAGE="Failed to start"
journalctl -g "error|failed|exception"

# Match field
journalctl _SYSTEMD_UNIT="nginx.service"

# Kernel messages
journalctl -k
journalctl --dmesg

# =============================================================================
# OUTPUT FORMATTING
# =============================================================================

# Short output (like syslog)
journalctl -o short

# Short-iso format
journalctl -o short-iso

# Export to JSON
journalctl -o json

# Pretty JSON
journalctl -o json-pretty

# Verbose (show all fields)
journalctl -o verbose

# Export to cat
journalctl -o cat

# =============================================================================
# MANIPULATION
# =============================================================================

# Follow (tail -f)
journalctl -f

# Show last N entries
journalctl -n 100

# Disk usage
journalctl --disk-usage

# Vacuum old logs
journalctl --vacuum-size=100M    # Keep last 100MB
journalctl --vacuum-time=2weeks  # Keep 2 weeks

# Vacuum specific time
journalctl --vacuum-files=5  # Keep 5 files

Journal Configuration

[Journal]
# Storage location
Storage=persistent            # persistent (disk), volatile (memory), auto, none
Compress=yes
Seal=yes

# Size limits
SystemMaxUse=500M
SystemKeepFree=500M
SystemMaxFileSize=50M
SystemMaxFiles=100

# Runtime limits
RuntimeMaxUse=100M
RuntimeKeepFree=100M

# Forwarding
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no

# Rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=1000

# Split user/root journals
SplitMode=uid

36.4 Remote Logging

Centralized Logging Architecture

                     Centralized Logging
+------------------------------------------------------------------+
|                                                                   |
|    Client Servers              Log Server                         |
|    +----------------+        +----------------+                   |
|    | web1          |        |                |                   |
|    | app1          |------->| rsyslogd      |                   |
|    | db1           |        |                |                   |
|    | +--------+   |        | +----------+  |                   |
|    | |rsyslog |---------->| |Storage   |  |                   |
|    | +--------+   |        | |Elastic    |  |                   |
|    +----------------+        | |search    |  |                   |
|                              +----------+  |                   |
|                                     |      |                   |
|                              +------v------+                   |
|                              | Kibana     |                   |
|                              +------------+                   |
|                                                                   |
+------------------------------------------------------------------+

Setting Up Remote Logging

# =============================================================================
# RSYSLOG SERVER - /etc/rsyslog.conf
# =============================================================================

# Enable TCP/UDP listeners
module(load="imudp")
module(load="imtcp")

# TCP listener
input(type="imtcp" port="514" ruleset="RemoteLogging")

# UDP listener
input(type="imudp" port="514" ruleset="RemoteLogging")

# Ruleset for remote logging
ruleset(name="RemoteLogging") {
    # Store by client hostname
    template(name="RemoteHost" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")

    # Create directory if needed
    action(type="omdir" dir="/var/log/remote/%HOSTNAME%")

    # Log to file
    action(type="omfile" file="?RemoteHost" template="RSYSLOG_FileFormat")

    # Also log to central file
    action(type="omfile" file="/var/log/remote-all.log" template="RSYSLOG_FileFormat")
}

# =============================================================================
# RSYSLOG CLIENT - /etc/rsyslog.conf
# =============================================================================

# Forward all to remote server
*.* @@logserver.example.com:514

# Forward with specific template
*.* @@logserver.example.com:514;RSYSLOG_SyslogProtocol23Format

# Selective forwarding
if $programname == "nginx" then @@logserver.example.com:514
& ~

# =============================================================================
# TLS-ENCRYPTED REMOTE LOGGING
# =============================================================================

# Server: Enable TLS
module(load="imtcp" tls.tlscfgcmd="")
input(type="imtcp" port="514" ruleset="RemoteLogging" tls="on")

# Client: Connect with TLS
action(type="omfwd" protocol="tcp" target="logserver.example.com" port="514" tls="on")

36.5 logrotate Deep Dive

logrotate Configuration

# /etc/logrotate.conf - Main Configuration

# Global settings
weekly                  # Rotate weekly
rotate 4                # Keep 4 weeks of logs
create                 # Create new empty file after rotation
compress              # Compress rotated logs
delaycompress         # Don't compress immediately (wait for next rotate)
dateext              # Use date suffix
dateformat -%Y%m%d   # Date format
maxage 365           # Delete logs older than 365 days

# Include additional configs
include /etc/logrotate.d

# Default settings for packages
/var/log/wtmp {
    monthly            # Rotate monthly
    create 0664 root utmp  # Permissions
    minsize 1M         # Minimum size
    rotate 1           # Keep 1 copy
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

logrotate for Applications

/var/log/nginx/*.log {
    daily               # Daily rotation
    missingok          # Don't error if missing
    rotate 14          # Keep 14 days
    compress           # Compress
    delaycompress      # Wait for next rotation
    notifempty        # Don't rotate if empty
    create 0640 www-data adm  # Permissions

    # Execute script after rotation
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript

    # Shared scripts run once per rotation
    sharedscripts
}

/etc/logrotate.d/apache2
/var/log/apache2/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 root adm
    sharedscripts

    postrotate
        /bin/systemctl reload apache2 > /dev/null 2>&1 || true
    endscript
}

/etc/logrotate.d/mysql
/var/log/mysql/*.log {
    daily
    rotate 7
    missingok
    create 660 mysql mysql
    compress
    postrotate
        test -x /usr/bin/mysqladmin || exit 0
        mysqladmin --silent flush-logs
    endscript
}

36.6 Exam Tips

Syslog facilities: Know 0-23 (kern, user, mail, daemon, auth, etc.)
Syslog priorities: Know 0-7 (emerg to debug)
rsyslog: Understand filters, templates, and output modules
journalctl: Know filtering by time, priority, unit, and field
journald config: Storage, size limits, forwarding settings
Remote logging: Understand UDP vs TCP vs RELP
logrotate: Know directives (rotate, compress, missingok, etc.)
Troubleshooting: Use journalctl for debugging
Centralized logging: Understand architecture
Security: Know log file permissions

Summary

In this chapter, you learned:

✅ Linux logging architecture overview
✅ Syslog facilities and priorities
✅ rsyslog configuration and filtering
✅ systemd journal and journalctl usage
✅ Remote logging setup
✅ logrotate configuration
✅ Log management best practices

Next Chapter

Chapter 37: Logrotate and Log Management

Last Updated: February 2026