Skip to content
Flowchart Learning

Journal_analysis

Chapter 86: Systemd Journal Analysis - Deep Dive

Section titled “Chapter 86: Systemd Journal Analysis - Deep Dive”

Mastering System Diagnostics with journalctl

Section titled “Mastering System Diagnostics with journalctl”

86.1 Understanding Systemd Journal

Section titled “86.1 Understanding Systemd Journal”

Journal Architecture

Section titled “Journal Architecture”
                     systemd Journal System
+------------------------------------------------------------------+
|                                                                   |
|    +-------------+     +-------------+     +-------------+        |
|    | Applications|     |   Kernel   |     |   Systemd   |        |
|    |  (syslog)  |     |  (/dev/kmsg)|     |  (journald) |        |
|    +-------------+     +-------------+     +-------------+        |
|           |               |               |                        |
|           v               v               v                        |
|    +-------------------------------------------------------------+ |
|    |                    journald Daemon                          | |
|    |                                                             | |
|    |  - Receives log messages                                   | |
|    |  - Indexes for fast searching                              | |
|    |  - Stores in binary format                                 | |
|    |  - Applies filtering                                       | |
|    +-----------------------------+-------------------------------+ |
|                    |                    |                          |
|                    v                    v                          |
|         /run/log/journal    /var/log/journal                     |
|         (volatile)           (persistent)                          |
|                                                                   |
+------------------------------------------------------------------+

Journal Storage

Section titled “Journal Storage”
                     Journal Storage Types
+------------------------------------------------------------------+
|                                                                   |
|    volatile                                                        |
|    +----------------------------------------------------------+   |
|    |  - Stored in /run/log/journal                           |   |
|    |  - Lost on reboot                                        |   |
|    |  - Default when /var not writable                       |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    persistent                                                     |
|    +----------------------------------------------------------+   |
|    |  - Stored in /var/log/journal                           |   |
|    |  - Persists across reboots                              |   |
|    |  - Configured in journald.conf                          |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    auto                                                            |
|    +----------------------------------------------------------+   |
|    |  - Uses persistent if /var is on persistent storage    |   |
|    |  - Uses volatile otherwise                               |   |
|    +----------------------------------------------------------+   |
|                                                                   |
|    none                                                            |
|    +----------------------------------------------------------+   |
|    |  - Doesn't store any logs                               |   |
|    |  - Forwards to syslog only                             |   |
|    +----------------------------------------------------------+   |
|                                                                   |
+------------------------------------------------------------------+

86.2 Basic journalctl Commands

Section titled “86.2 Basic journalctl Commands”

Viewing Logs

Section titled “Viewing Logs”
Terminal window
# =============================================================================
# BASIC VIEWING
# =============================================================================


# View all logs (newest first)
journalctl


# View all logs (oldest first)
journalctl -r


# View kernel messages
journalctl -k
journalctl --dmesg


# View current boot
journalctl -b


# View previous boot
journalctl -b -1


# View specific boot ID
journalctl -b abc123def456


# =============================================================================
# TIME-BASED FILTERING
# =============================================================================


# Since specific time
journalctl --since "2024-01-01 00:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday
journalctl --since "2 days ago"


# Until specific time
journalctl --until "2024-01-01 12:00:00"
journalctl --until "1 hour ago"


# Time range
journalctl --since "2024-01-01 10:00:00" --until "2024-01-01 11:00:00"


# =============================================================================
# PRIORITY FILTERING
# =============================================================================


# Priority levels: emerg(0) > alert(1) > crit(2) > err(3) > warning(4) > notice(5) > info(6) > debug(7)


# Show error and above
journalctl -p err
journalctl -p 3


# Show warning and above
journalctl -p warning


# Show multiple priorities
journalctl -p err..warning


# All priorities
journalctl -p 0..7

Unit/Service Filtering

Section titled “Unit/Service Filtering”
Terminal window
# =============================================================================
# SERVICE FILTERING
# =============================================================================


# Single service
journalctl -u nginx.service


# Multiple services
journalctl -u nginx.service -u mysql.service


# Follow service logs
journalctl -u nginx.service -f


# Since service start
journalctl -u nginx.service -b


# Failed service
journalctl --failed


# All failed units
systemctl --failed


# =============================================================================
# PROCESS FILTERING
# =============================================================================


# By PID
journalctl _PID=1234


# By UID
journalctl _UID=1000
journalctl _UID=$(id -u username)


# By GID
journalctl _GID=1000


# By executable
journalctl _EXE=/usr/bin/nginx


# By command line
journalctl _CMDLINE="nginx -g daemon off"


# =============================================================================
# KERNEL MESSAGES
# =============================================================================


# Kernel messages only
journalctl -k
journalctl --dmesg


# With kernel ring buffer size
journalctl -k --cursor-file=/var/log/journal/boot-id


# =============================================================================
# BOOT SPECIFIC
# =============================================================================


# List boots
journalctl --list-boots


# Current boot
journalctl -b


# Previous boot
journalctl -b -1


# N boots ago
journalctl -b -2


# Specific boot ID
journalctl -b abc123

86.3 Advanced Filtering

Section titled “86.3 Advanced Filtering”

Complex Filters

Section titled “Complex Filters”
Terminal window
# =============================================================================
# MESSAGE CONTENT FILTERING
# =============================================================================


# Simple match
journalctl MESSAGE="Failed to start"


# Regex match
journalctl -g "error|failed|exception"


# Contains substring
journalctl MESSAGE_STRIPAS=true | grep -i error


# =============================================================================
# FIELD MATCHING
# =============================================================================


# Match specific field
journalctl _SYSTEMD_UNIT="nginx.service"


# Negate match
journalctl _SYSTEMD_UNIT!="nginx.service"


# Multiple conditions (AND)
journalctl _SYSTEMD_UNIT="nginx.service" _PID=1234


# Multiple conditions (OR)
journalctl + _SYSTEMD_UNIT="httpd.service"


# =============================================================================
# FACILITY/SYSLOG
# =============================================================================


# Based on syslog facility (via forwarding)
journalctl SYSLOG_FACILITY=3
journalctl SYSLOG_FACILITY=daemon


# Based on syslog identifier
journalctl SYSLOG_IDENTIFIER=systemd


# =============================================================================
# HOST/TRANSPORT
# =============================================================================


# Remote boots
journalctl _HOSTNAME=server1.example.com


# Specific boot ID
journalctl _BOOT_ID=abc123


# Transport method
journalctl _TRANSPORT=kernel
journalctl _TRANSPORT=syslog
journalctl _TRANSPORT=journal

86.4 Output Formatting

Section titled “86.4 Output Formatting”

Display Options

Section titled “Display Options”
Terminal window
# =============================================================================
# OUTPUT FORMATS
# =============================================================================


# Short format (default)
journalctl -o short


# Short with ISO timestamp
journalctl -o short-iso


# Short with full timestamp
journalctl -o short-full


# Short with monotonic clock
journalctl -o short-monotonic


# UTC time
journalctl -o short-precise


# Verbose (all fields)
journalctl -o verbose


# JSON
journalctl -o json


# Pretty JSON
journalctl -o json-pretty


# JSON binary fields
journalctl -o json-sse


# Export to cat
journalctl -o cat


# Export to export (binary)
journalctl -o export


# =============================================================================
# CUSTOM FORMAT
# =============================================================================


# Custom fields
journalctl -o format '{{.HOSTNAME}} {{.MESSAGE}}'


# Full format
journalctl -o short-full


# =============================================================================
# HEAD/TAIL
# =============================================================================


# Last N entries
journalctl -n 100


# Follow with last N
journalctl -f -n 50

86.5 Real-time Monitoring

Section titled “86.5 Real-time Monitoring”

Live Log Monitoring

Section titled “Live Log Monitoring”
Terminal window
# =============================================================================
# FOLLOW MODE
# =============================================================================


# Follow all logs
journalctl -f


# Follow specific service
journalctl -u nginx.service -f


# Follow kernel messages
journalctl -k -f


# =============================================================================
# MONITORING
# =============================================================================


# Monitor for new entries (like tail -f)
journalctl -f


# Monitor with priority
journalctl -p err -f


# Watch specific unit
watch -n 1 'journalctl -u nginx.service -n 10 --no-pager'


# =============================================================================
# ALERTING
# =============================================================================


# Watch for errors
journalctl -p err -f | while read line; do
    echo "ERROR: $line" | mail -s "Server Error" admin@example.com
done

86.6 Maintenance and Troubleshooting

Section titled “86.6 Maintenance and Troubleshooting”

Disk Usage

Section titled “Disk Usage”
Terminal window
# =============================================================================
# DISK USAGE
# =============================================================================


# Check disk usage
journalctl --disk-usage


# Current journal size
du -sh /var/log/journal/


# Per-user usage
journalctl --user --disk-usage


# =============================================================================
# CLEANUP
# =============================================================================


# Keep only last 100MB
journalctl --vacuum-size=100M


# Keep only last 2 weeks
journalctl --vacuum-time=2weeks


# Keep only 5 files
journalctl --vacuum-files=5


# All users
journalctl --user --vacuum-size=50M


# =============================================================================
# ROTATION
# =============================================================================


# Force rotation
journalctl --rotate


# Archive old entries
journalctl --archive

Troubleshooting Examples

Section titled “Troubleshooting Examples”
Terminal window
# =============================================================================
# SERVICE FAILURE
# =============================================================================


# Check service failure
journalctl -u nginx.service --failed


# Last 10 lines before failure
journalctl -u nginx.service -b | tail -50


# Since last successful start
journalctl -u nginx.service --since "$(systemctl show -p ActiveEnterTimestamp nginx.service --value)"


# =============================================================================
# BOOT ISSUES
# =============================================================================


# Show boot errors
journalctl -b -p err


# Previous boot issues
journalctl -b -1 -p err


# Kernel failures
journalctl -k --priority=err


# =============================================================================
# PERFORMANCE ISSUES
# =============================================================================


# High CPU by service
journalctl -u nginx.service | awk '{print $5}' | sort | uniq -c | sort -rn | head


# Slow boot analysis
systemd-analyze blame


# Critical chain
systemd-analyze critical-chain

86.7 Configuration

Section titled “86.7 Configuration”

journald Configuration

Section titled “journald Configuration”
/etc/systemd/journald.conf
[Journal]
# Storage location
Storage=persistent
# Options: auto, volatile, persistent, none


# Size limits
SystemMaxUse=500M
SystemKeepFree=500M
SystemMaxFileSize=50M
SystemMaxFiles=100


# Runtime limits
RuntimeMaxUse=100M
RuntimeKeepFree=100M
RuntimeMaxFileSize=10M
RuntimeMaxFiles=3


# Compression
Compress=yes
Seal=yes


# Forwarding
ForwardToSyslog=yes
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes


# Rate limiting
RateLimitIntervalSec=30s
RateLimitBurst=1000


# Split mode
SplitMode=uid

86.8 Exam Tips

Section titled “86.8 Exam Tips”

Important

  1. Filtering: Know —since, —until, -u, -p, -g flags
  2. Boot logs: Use -b, -b -1 for previous boot
  3. Priority: -p err (0-3)
  4. Real-time: Use -f for follow mode
  5. Output: -o json-pretty for structured data
  6. Maintenance: —vacuum-size, —vacuum-time
  7. Systemd-analyze: For boot performance
  8. Field matching: _SYSTEMD_UNIT, _PID, etc.
  9. Disk usage: Check journal size regularly
  10. Rotation: Automatic in systemd

Summary

Section titled “Summary”

In this chapter, you learned:

  • ✅ Journal architecture and storage
  • ✅ Basic journalctl commands
  • ✅ Time and priority filtering
  • ✅ Unit and process filtering
  • ✅ Output formatting
  • ✅ Real-time monitoring
  • ✅ Maintenance and cleanup
  • ✅ Configuration options
  • ✅ Troubleshooting techniques

Next Chapter

Section titled “Next Chapter”

Chapter 87: strace, ltrace, and syscalls

Last Updated: February 2026