Journal Analysis

Chapter 86: Systemd Journal Analysis - Deep Dive

Mastering System Diagnostics with journalctl

Why This Matters in DevOps/SRE

Systemd journal is the primary logging mechanism in modern Linux systems:

Incident Response: First place to look when systems fail
Debugging: Understand application behavior and failures
Compliance: Audit logs for security and regulatory requirements
Performance: Identify bottlenecks and resource issues
Root Cause Analysis: Correlate events across services

Every DevOps and SRE must master journalctl for effective troubleshooting.

86.1 Understanding Systemd Journal

Journal Architecture

                     systemd Journal System
+------------------------------------------------------------------+
|                                                                   |
|    +-------------+     +-------------+     +-------------+        |
|    | Applications|     |   Kernel   |     |   Systemd   |        |
|    |  (syslog)  |     |  (/dev/kmsg)|     |  (journald) |        |
|    +-------------+     +-------------+     +-------------+        |
|           |               |               |                        |
|           v               v               v                        |
|    +-------------------------------------------------------------+ |
|    |                    journald Daemon                          | |
|    |                                                             | |
|    |  - Receives log messages                                   | |
|    |  - Indexes for fast searching                              | |
|    |  - Stores in binary format                                 | |
|    |  - Applies filtering                                       | |
|    +-----------------------------+-------------------------------+ |
|                    |                    |                          |
|                    v                    v                          |
|         /run/log/journal    /var/log/journal                     |
|         (volatile)           (persistent)                          |
|                                                                   |
+------------------------------------------------------------------+

Journal Storage

                     Journal Storage Types
+------------------------------------------------------------------+
|                                                                   |
|    volatile                                                        |
|    +----------------------------------------------------------+   |
|    |  - Stored in /run/log/journal                           |   |
|    |  - Lost on reboot                                        |   |
|    |  - Default when /var not writable                       |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    persistent                                                     |
|    +----------------------------------------------------------+   |
|    |  - Stored in /var/log/journal                           |   |
|    |  - Persists across reboots                              |   |
|    |  - Configured in journald.conf                          |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    auto                                                            |
|    +----------------------------------------------------------+   |
|    |  - Uses persistent if /var is on persistent storage    |   |
|    |  - Uses volatile otherwise                               |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    none                                                            |
|    +----------------------------------------------------------+   |
|    |  - Doesn't store any logs                               |   |
|    |  - Forwards to syslog only                             |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

86.2 Basic journalctl Commands

Viewing Logs

# =============================================================================
# BASIC VIEWING
# =============================================================================

# View all logs (newest first)
journalctl

# View all logs (oldest first)
journalctl -r

# View kernel messages
journalctl -k
journalctl --dmesg

# View current boot
journalctl -b

# View previous boot
journalctl -b -1

# View specific boot ID
journalctl -b abc123def456

# =============================================================================
# TIME-BASED FILTERING
# =============================================================================

# Since specific time
journalctl --since "2024-01-01 00:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday
journalctl --since "2 days ago"

# Until specific time
journalctl --until "2024-01-01 12:00:00"
journalctl --until "1 hour ago"

# Time range
journalctl --since "2024-01-01 10:00:00" --until "2024-01-01 11:00:00"

# =============================================================================
# PRIORITY FILTERING
# =============================================================================

# Priority levels: emerg(0) > alert(1) > crit(2) > err(3) > warning(4) > notice(5) > info(6) > debug(7)

# Show error and above
journalctl -p err
journalctl -p 3

# Show warning and above
journalctl -p warning

# Show multiple priorities
journalctl -p err..warning

# All priorities
journalctl -p 0..7

Unit/Service Filtering

# =============================================================================
# SERVICE FILTERING
# =============================================================================

# Single service
journalctl -u nginx.service

# Multiple services
journalctl -u nginx.service -u mysql.service

# Follow service logs
journalctl -u nginx.service -f

# Since service start
journalctl -u nginx.service -b

# Failed service
journalctl --failed

# All failed units
systemctl --failed

# =============================================================================
# PROCESS FILTERING
# =============================================================================

# By PID
journalctl _PID=1234

# By UID
journalctl _UID=1000
journalctl _UID=$(id -u username)

# By GID
journalctl _GID=1000

# By executable
journalctl _EXE=/usr/bin/nginx

# By command line
journalctl _CMDLINE="nginx -g daemon off"

# =============================================================================
# KERNEL MESSAGES
# =============================================================================

# Kernel messages only
journalctl -k
journalctl --dmesg

# With kernel ring buffer size
journalctl -k --cursor-file=/var/log/journal/boot-id

# =============================================================================
# BOOT SPECIFIC
# =============================================================================

# List boots
journalctl --list-boots

# Current boot
journalctl -b

# Previous boot
journalctl -b -1

# N boots ago
journalctl -b -2

# Specific boot ID
journalctl -b abc123

86.3 Advanced Filtering

Complex Filters

# =============================================================================
# MESSAGE CONTENT FILTERING
# =============================================================================

# Simple match
journalctl MESSAGE="Failed to start"

# Regex match
journalctl -g "error|failed|exception"

# Contains substring
journalctl MESSAGE_STRIPAS=true | grep -i error

# =============================================================================
# FIELD MATCHING
# =============================================================================

# Match specific field
journalctl _SYSTEMD_UNIT="nginx.service"

# Negate match
journalctl _SYSTEMD_UNIT!="nginx.service"

# Multiple conditions (AND)
journalctl _SYSTEMD_UNIT="nginx.service" _PID=1234

# Multiple conditions (OR)
journalctl + _SYSTEMD_UNIT="httpd.service"

# =============================================================================
# FACILITY/SYSLOG
# =============================================================================

# Based on syslog facility (via forwarding)
journalctl SYSLOG_FACILITY=3
journalctl SYSLOG_FACILITY=daemon

# Based on syslog identifier
journalctl SYSLOG_IDENTIFIER=systemd

# =============================================================================
# HOST/TRANSPORT
# =============================================================================

# Remote boots
journalctl _HOSTNAME=server1.example.com

# Specific boot ID
journalctl _BOOT_ID=abc123

# Transport method
journalctl _TRANSPORT=kernel
journalctl _TRANSPORT=syslog
journalctl _TRANSPORT=journal

86.4 Output Formatting

Display Options

# =============================================================================
# OUTPUT FORMATS
# =============================================================================

# Short format (default)
journalctl -o short

# Short with ISO timestamp
journalctl -o short-iso

# Short with full timestamp
journalctl -o short-full

# Short with monotonic clock
journalctl -o short-monotonic

# UTC time
journalctl -o short-precise

# Verbose (all fields)
journalctl -o verbose

# JSON
journalctl -o json

# Pretty JSON
journalctl -o json-pretty

# JSON binary fields
journalctl -o json-sse

# Export to cat
journalctl -o cat

# Export to export (binary)
journalctl -o export

# =============================================================================
# CUSTOM FORMAT
# =============================================================================

# Custom fields
journalctl -o format '{{.HOSTNAME}} {{.MESSAGE}}'

# Full format
journalctl -o short-full

# =============================================================================
# HEAD/TAIL
# =============================================================================

# Last N entries
journalctl -n 100

# Follow with last N
journalctl -f -n 50

86.5 Real-time Monitoring

Live Log Monitoring

# =============================================================================
# FOLLOW MODE
# =============================================================================

# Follow all logs
journalctl -f

# Follow specific service
journalctl -u nginx.service -f

# Follow kernel messages
journalctl -k -f

# =============================================================================
# MONITORING
# =============================================================================

# Monitor for new entries (like tail -f)
journalctl -f

# Monitor with priority
journalctl -p err -f

# Watch specific unit
watch -n 1 'journalctl -u nginx.service -n 10 --no-pager'

# =============================================================================
# ALERTING
# =============================================================================

# Watch for errors
journalctl -p err -f | while read line; do
    echo "ERROR: $line" | mail -s "Server Error" admin@example.com
done

86.6 Maintenance and Troubleshooting

Disk Usage

# =============================================================================
# DISK USAGE
# =============================================================================

# Check disk usage
journalctl --disk-usage

# Current journal size
du -sh /var/log/journal/

# Per-user usage
journalctl --user --disk-usage

# =============================================================================
# CLEANUP
# =============================================================================

# Keep only last 100MB
journalctl --vacuum-size=100M

# Keep only last 2 weeks
journalctl --vacuum-time=2weeks

# Keep only 5 files
journalctl --vacuum-files=5

# All users
journalctl --user --vacuum-size=50M

# =============================================================================
# ROTATION
# =============================================================================

# Force rotation
journalctl --rotate

# Archive old entries
journalctl --archive

Troubleshooting Examples

# =============================================================================
# SERVICE FAILURE
# =============================================================================

# Check service failure
journalctl -u nginx.service --failed

# Last 10 lines before failure
journalctl -u nginx.service -b | tail -50

# Since last successful start
journalctl -u nginx.service --since "$(systemctl show -p ActiveEnterTimestamp nginx.service --value)"

# =============================================================================
# BOOT ISSUES
# =============================================================================

# Show boot errors
journalctl -b -p err

# Previous boot issues
journalctl -b -1 -p err

# Kernel failures
journalctl -k --priority=err

# =============================================================================
# PERFORMANCE ISSUES
# =============================================================================

# High CPU by service
journalctl -u nginx.service | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Slow boot analysis
systemd-analyze blame

# Critical chain
systemd-analyze critical-chain

86.7 Configuration

journald Configuration

[Journal]
# Storage location
Storage=persistent
# Options: auto, volatile, persistent, none

# Size limits
SystemMaxUse=500M
SystemKeepFree=500M
SystemMaxFileSize=50M
SystemMaxFiles=100

# Runtime limits
RuntimeMaxUse=100M
RuntimeKeepFree=100M
RuntimeMaxFileSize=10M
RuntimeMaxFiles=3

# Compression
Compress=yes
Seal=yes

# Forwarding
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes

# Rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=1000

# Split mode
SplitMode=uid

86.8 Exam Tips

Filtering: Know —since, —until, -u, -p, -g flags
Boot logs: Use -b, -b -1 for previous boot
Priority: -p err (0-3)
Real-time: Use -f for follow mode
Output: -o json-pretty for structured data
Maintenance: —vacuum-size, —vacuum-time
Systemd-analyze: For boot performance
Field matching: _SYSTEMD_UNIT, _PID, etc.
Disk usage: Check journal size regularly
Rotation: Automatic in systemd

Common Mistakes & Anti-Patterns

1. Not Using Proper Time Ranges

WRONG:

# Getting all logs - overwhelming and slow
journalctl

CORRECT:

# Use time filters
journalctl --since "1 hour ago"
journalctl --since "2024-01-15 10:00:00" --until "2024-01-15 11:00:00"
journalctl -S -24h

Why: Without time filters, you get massive output and slow performance.

2. Ignoring Priority Levels

WRONG:

# Showing all priorities
journalctl -m

CORRECT:

# Filter by priority (0=emergency to 7=debug)
journalctl -p err        # Errors and above
journalctl -p warning   # Warnings and above
journalctl -p 0..3      # Emergency to Error

Why: Focus on relevant logs to find issues faster.

3. Not Using Unit Filters

WRONG:

# Searching everywhere
journalctl | grep nginx

CORRECT:

# Filter by service unit
journalctl -u nginx.service
journalctl -u nginx.service --since "1 hour ago"
journalctl -u nginx.service -u php-fpm.service

Why: Much faster and more relevant results.

4. Forgetting About Disk Space

WRONG:

# Not checking journal size
df -h  # See disk full!

CORRECT:

# Check journal size
journalctl --disk-usage
du -sh /var/log/journal

# Clean old logs
journalctl --vacuum-time=7d
journalctl --vacuum-size=500M

Why: Journal can grow large and fill disk.

5. Not Using Real-time Monitoring

WRONG:

# Checking logs periodically
while true; do journalctl -n 20; sleep 60; done

CORRECT:

# Use built-in real-time monitoring
journalctl -f
journalctl -f -u nginx.service
journalctl -f -p err

Why: Built-in follow mode is more efficient and handles rotation.

Summary

In this chapter, you learned:

✅ Journal architecture and storage
✅ Basic journalctl commands
✅ Time and priority filtering
✅ Unit and process filtering
✅ Output formatting
✅ Real-time monitoring
✅ Maintenance and cleanup
✅ Configuration options
✅ Troubleshooting techniques

Next Chapter

Chapter 87: strace, ltrace, and syscalls

Last Updated: February 2026