Skip to content

systemd: Units, Targets, Journals & Internals

  • Master systemd unit types, dependencies, and security features
  • Understand the journal system and structured logging
  • Know how to write production-grade systemd services
  • Understand systemd timers as a cron replacement
  • Troubleshoot complex systemd dependency issues

systemd is the master manager of a Linux system. It’s the very first program that starts (PID 1) and its job is to start all the other background services (like web servers and databases), keep them running, and manage their logs.

systemd PID 1 — The System Manager
─────────────────────────────────────
┌────────────────────────────────────────────────────────────┐
│ systemd (PID=1) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ Unit Engine │ │ D-Bus IPC │ │ Job Queue │ │
│ │ │ │ │ │ │ │
│ │ Dependency │ │ API for │ │ Pending: │ │
│ │ resolver │ │ systemctl, │ │ - start nginx │ │
│ │ State mgr │ │ loginctl │ │ - stop redis │ │
│ └──────────────┘ └──────────────┘ └─────────────────┘ │
│ │
│ Satellite Daemons: │
│ journald udevd networkd resolved logind │
└────────────────────────────────────────────────────────────┘
systemctl sends commands via D-Bus
journalctl reads from journald socket

Unit files have three sections: [Unit], [Service/Timer/Socket/...], and [Install].

[Unit]
# ─── Description ────────────────────────────────────────────
Description=Short description shown in systemctl status
Documentation=https://docs.example.com man:nginx(8)
# Multiple Documentation= entries allowed
# ─── Ordering ───────────────────────────────────────────────
Before=unit1.service unit2.service # This unit starts BEFORE these
After=network.target syslog.target # This unit starts AFTER these
# ─── Dependencies ────────────────────────────────────────────
Requires=postgresql.service # Hard dep: if postgres fails, we fail
Wants=redis.service # Soft dep: start redis if available
BindsTo=container@.service # Stronger: stop if dep stops
PartOf=group.target # Stop/restart when parent does
# ─── Conditions ──────────────────────────────────────────────
ConditionPathExists=/etc/myapp/config.yaml # Only start if file exists
ConditionFileIsExecutable=/opt/myapp/server # Only if binary exists
ConditionKernelVersion=>=5.10 # Min kernel version
ConditionVirtualization=no # Only on bare metal
ConditionHost=prod-server-01 # Only on specific host
# ─── Assertions (vs Conditions) ──────────────────────────────
# Conditions: if fails, unit is skipped (not counted as failure)
# Assertions: if fails, unit fails (counts as error)
AssertPathExists=/var/lib/myapp # Must exist or FAIL
[Service]
# ─── Service Type ────────────────────────────────────────────
# simple: ExecStart is the main process (PID tracked)
# forking: ExecStart forks; parent exits; child becomes main
# notify: Like simple, but app sends sd_notify() when ready
# oneshot: ExecStart exits when done; service stays "active"
# dbus: Service is ready when D-Bus name acquired
# idle: Like simple, but delayed until job queue empty
Type=notify
# ─── Commands ─────────────────────────────────────────────────
ExecStartPre=/usr/bin/test -f /etc/myapp/config.yaml # Pre-start check
ExecStart=/opt/myapp/server --config /etc/myapp/config.yaml
ExecStartPost=/usr/bin/curl -s localhost:8080/health # Post-start check
ExecReload=/bin/kill -HUP $MAINPID # Graceful reload
ExecStop=/bin/kill -TERM $MAINPID # Graceful stop
ExecStopPost=/usr/bin/rm -f /run/myapp.pid # Cleanup
# ─── Process Identity ─────────────────────────────────────────
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
RootDirectory=/opt/myapp # chroot (advanced)
# ─── Environment ──────────────────────────────────────────────
Environment=NODE_ENV=production PORT=8080
EnvironmentFile=-/etc/myapp/env # - prefix = ignore if missing
PassEnvironment=HOME LANG # Pass from systemd's environment
# ─── PID File (for Type=forking only) ─────────────────────────
PIDFile=/run/myapp.pid
# ─── Restart Policy ───────────────────────────────────────────
Restart=on-failure # no|always|on-success|on-failure|on-abnormal|on-abort
RestartSec=10 # Wait 10s before restart
StartLimitIntervalSec=300 # Rate limit window (5 minutes)
StartLimitBurst=5 # Max 5 starts in that window
# ─── Timeouts ─────────────────────────────────────────────────
TimeoutStartSec=30 # Fail if not ready in 30s
TimeoutStopSec=30 # SIGKILL if not stopped in 30s
TimeoutAbortSec=10 # After abort before SIGKILL
WatchdogSec=30 # App must send keepalive every 30s
# ─── Resource Limits ──────────────────────────────────────────
# Cgroup-based (v2):
MemoryMax=2G # Hard limit (OOM kill)
MemoryHigh=1.5G # Soft limit (throttle)
CPUQuota=200% # Max 2 CPUs
CPUWeight=100 # Relative CPU priority
IOWeight=100 # Relative I/O priority
TasksMax=1000 # Max PIDs/threads
# ulimit-based:
LimitNOFILE=65536 # Max file descriptors
LimitNPROC=4096 # Max processes
LimitCORE=infinity # Core dump size (infinity = no limit)
# ─── Security Hardening ───────────────────────────────────────
NoNewPrivileges=yes # Cannot gain privileges via setuid/setgid
PrivateTmp=yes # Isolated /tmp
PrivateDevices=yes # No access to /dev devices
PrivateNetwork=no # (yes = no network access)
PrivateUsers=no # (yes = empty user/group database)
ProtectSystem=strict # /usr, /boot, /etc read-only
ProtectSystem=full # /usr, /boot read-only
ProtectHome=yes # /home, /root, /run/user inaccessible
ProtectKernelModules=yes # Cannot load kernel modules
ProtectKernelTunables=yes # Cannot modify kernel parameters
ProtectKernelLogs=yes # Cannot access kernel log
ProtectClock=yes # Cannot change system time
ProtectControlGroups=yes # Cgroup filesystem read-only
RestrictNamespaces=yes # Cannot create namespaces
RestrictSUIDSGID=yes # Cannot create setuid/setgid files
LockPersonality=yes # Cannot change execution domain
# Allow specific paths:
ReadWritePaths=/var/lib/myapp /var/log/myapp
ReadOnlyPaths=/etc/myapp
InaccessiblePaths=/boot /root
# Capabilities:
CapabilityBoundingSet=CAP_NET_BIND_SERVICE # Only this cap allowed
AmbientCapabilities=CAP_NET_BIND_SERVICE # Grant to non-root process
# Syscall filtering:
SystemCallFilter=@system-service # Common safe syscalls
SystemCallFilter=~@debug @mount @reboot # Block dangerous groups
SystemCallArchitectures=native # Only native arch syscalls
# OOM priority:
OOMScoreAdjust=-900 # Protect from OOM killer
[Install]
# Defines which targets enable this unit
WantedBy=multi-user.target # Normal service
WantedBy=graphical.target # GUI service
# Alternative targets
RequiredBy=emergency.target # Must be active for emergency mode
# Alias names for this unit
Alias=myapp.service myapplication.service
# Units that should be started if this one is started
Also=myapp-monitor.service myapp-cleanup.timer

systemd timers are more powerful than cron: they support boot-relative timing, catch up on missed runs, integrate with the journal, and have dependencies.

/etc/systemd/system/backup.timer
[Unit]
Description=Daily backup at 2 AM
[Timer]
# ─── Calendar-based (like cron) ──────────────────────────────
OnCalendar=*-*-* 02:00:00 # Daily at 2 AM
OnCalendar=Mon..Fri 09:00:00 # Weekdays at 9 AM
OnCalendar=Sat,Sun 10:00:00 # Weekends at 10 AM
OnCalendar=*-*-01 00:00:00 # First of every month
# ─── Monotonic (relative to boot or last run) ─────────────────
OnBootSec=5min # 5 minutes after boot
OnStartupSec=10min # 10 minutes after systemd start
OnUnitActiveSec=1h # 1 hour after last successful run
OnUnitInactiveSec=30min # 30 minutes after last run ended
# ─── Options ─────────────────────────────────────────────────
Persistent=true # Run immediately if missed while off
RandomizedDelaySec=300 # Add random delay up to 5 min (avoid herd)
AccuracySec=1min # Trigger within 1 minute of scheduled time
Unit=backup.service # Which service to activate
[Install]
WantedBy=timers.target
/etc/systemd/system/backup.service
[Unit]
Description=Backup Service
[Service]
Type=oneshot # Run and exit (timer controls scheduling)
User=backup
ExecStart=/usr/local/bin/backup.sh
Terminal window
# Timer management
systemctl list-timers --all # Show all timers and next run time
systemctl status backup.timer # Timer status
systemctl start backup.service # Run manually right now
journalctl -u backup.service -n 20 # Last 20 runs' logs
# Equivalent cron expressions:
# cron: "0 2 * * *" = OnCalendar=*-*-* 02:00:00
# cron: "*/5 * * * *" = OnUnitActiveSec=5min

The systemd journal (journald) is the logging daemon for systemd. It collects logs from all services, the kernel, and can forward to syslog.

Journal Architecture
─────────────────────
Applications:
┌──────────┐ ┌──────────┐ ┌──────────┐
│ nginx │ │ postgres │ │ kernel │
└─────┬────┘ └─────┬────┘ └─────┬────┘
│ │ │
│ stdout/stderr │ /dev/kmsg
│ │ │
┌─────▼─────────────▼──────────────▼─────┐
│ journald │
│ Structured storage: /var/log/journal/ │
│ Each entry has: timestamp, PID, UID, │
│ service name, priority, message, etc. │
└──────────────────┬──────────────────────┘
┌────────┼────────┐
▼ ▼ ▼
journalctl rsyslog ElasticSearch
(local view)(forward) (via filebeat)
Terminal window
# ─── Basic Usage ──────────────────────────────────────────
journalctl # All journal entries
journalctl -f # Follow (like tail -f)
journalctl -n 50 # Last 50 lines
journalctl -r # Reverse order (newest first)
# ─── By Service ───────────────────────────────────────────
journalctl -u nginx # All nginx logs
journalctl -u nginx -u postgresql # Multiple services
journalctl -u nginx -f # Follow nginx logs
# ─── By Priority ──────────────────────────────────────────
journalctl -p err # Errors only
journalctl -p warning # Warnings and above
journalctl -p 0..3 # emerg, alert, crit, err
# Priorities: 0=emerg 1=alert 2=crit 3=err 4=warning 5=notice 6=info 7=debug
# ─── By Time ──────────────────────────────────────────────
journalctl --since "2024-01-15 10:00:00"
journalctl --since "1 hour ago"
journalctl --since "today"
journalctl --since "2024-01-15" --until "2024-01-16"
journalctl -b # Current boot only
journalctl -b -1 # Previous boot
journalctl --list-boots # List all boots
# ─── Output Formats ───────────────────────────────────────
journalctl -u nginx -o json # JSON format
journalctl -u nginx -o json-pretty # Pretty JSON
journalctl -u nginx -o cat # Message only (no metadata)
journalctl -u nginx -o verbose # All fields
# ─── Kernel Messages ──────────────────────────────────────
journalctl -k # Kernel messages only
journalctl -k -b # Kernel messages, current boot
journalctl -k --since "1 hour ago" | grep -i error
# ─── Advanced Filtering ───────────────────────────────────
journalctl _SYSTEMD_UNIT=nginx.service
journalctl _PID=1234 # Specific PID
journalctl _UID=1000 # Specific user
journalctl _COMM=bash # Specific executable name
# ─── Storage Management ───────────────────────────────────
journalctl --disk-usage # Journal disk usage
journalctl --vacuum-size=1G # Keep only 1GB of logs
journalctl --vacuum-time=7d # Keep only last 7 days
/etc/systemd/journald.conf
[Journal]
Storage=persistent # persistent (disk), volatile (RAM), auto, none
Compress=yes # Compress stored data
MaxRetentionSec=30day # Delete entries older than 30 days
MaxFileSec=1week # Rotate journal files weekly
SystemMaxUse=2G # Max disk space for system journal
SystemKeepFree=1G # Always keep 1G free
SystemMaxFileSize=200M # Max size per journal file
RuntimeMaxUse=100M # Max RAM for volatile journal
ForwardToSyslog=no # Forward to syslog (if rsyslog is running)
ForwardToKMsg=no # Forward to kernel message buffer
ForwardToConsole=no # Forward to console (for debug)
MaxLevelConsole=info # Min priority to show on console
RateLimitIntervalSec=30s # Rate limiting window
RateLimitBurst=10000 # Max messages per window

Services start on-demand when a connection arrives:

/etc/systemd/system/myapp.socket
[Unit]
Description=My App Socket
[Socket]
ListenStream=8080 # TCP port
Accept=no # Pass the socket (not a new FD per connection)
[Install]
WantedBy=sockets.target
/etc/systemd/system/myapp.service
[Unit]
Description=My App
Requires=myapp.socket # Activated by socket
[Service]
ExecStart=/opt/myapp/server
StandardInput=socket # Receive the listening socket from systemd
Terminal window
# Systemd holds the socket open even when service is stopped
# First connection → systemd passes socket → service starts
systemctl enable myapp.socket
systemctl start myapp.socket
# Service is NOT running yet
curl localhost:8080
# This triggers myapp.service to start, then serves the request

Drop-in Files (Override without Modifying Original)

Section titled “Drop-in Files (Override without Modifying Original)”
Terminal window
# Edit vendor unit files without touching them:
# Creates /etc/systemd/system/nginx.service.d/override.conf
systemctl edit nginx
# Or manually:
mkdir -p /etc/systemd/system/nginx.service.d/
cat > /etc/systemd/system/nginx.service.d/override.conf << 'EOF'
[Service]
# Add environment variable without touching vendor unit
Environment=NGINX_ENTRYPOINT_QUIET_LOGS=1
# Increase memory limit
MemoryMax=2G
# Add pre-start hook
ExecStartPre=/usr/local/bin/pre-nginx-check.sh
EOF
systemctl daemon-reload
systemctl restart nginx

7.6 Production Runbook: systemd Service Management

Section titled “7.6 Production Runbook: systemd Service Management”
Terminal window
# ── Service Lifecycle ─────────────────────────────────────
systemctl start nginx # Start immediately
systemctl stop nginx # Stop gracefully (SIGTERM → SIGKILL)
systemctl restart nginx # Stop then start
systemctl reload nginx # Reload config without restart (SIGHUP)
systemctl reload-or-restart nginx # Reload if supported, else restart
systemctl condrestart nginx # Restart only if currently running
# ── Enable/Disable ────────────────────────────────────────
systemctl enable nginx # Enable on boot
systemctl disable nginx # Disable from boot
systemctl enable --now nginx # Enable AND start now
systemctl disable --now nginx # Disable AND stop now
systemctl is-enabled nginx # Check if enabled
# ── Status and Inspection ─────────────────────────────────
systemctl status nginx # Status + last 10 log lines
systemctl is-active nginx # active/inactive/failed
systemctl is-failed nginx # true/false
# ── Dependency Analysis ───────────────────────────────────
systemctl list-dependencies nginx # What nginx depends on
systemctl list-dependencies --reverse nginx # What depends on nginx
systemctl list-dependencies --before nginx # Units that start before
systemctl list-dependencies --after nginx # Units that start after
# ── Unit Files ────────────────────────────────────────────
systemctl cat nginx # Show effective unit file
systemctl show nginx # All properties (machine-readable)
systemctl edit nginx # Edit/create override
# ── Failed Services ───────────────────────────────────────
systemctl --failed # List all failed units
systemctl reset-failed # Clear failed state (allow retry)
systemctl reset-failed nginx # Clear failed state for nginx only
# ── System Power ──────────────────────────────────────────
systemctl poweroff # Power down
systemctl reboot # Reboot
systemctl suspend # Suspend (save to RAM)
systemctl hibernate # Hibernate (save to disk)

Terminal window
# Issue: Service keeps restarting
systemctl status myapp
journalctl -u myapp -n 50
# Check restart count
systemctl show myapp | grep NRestarts
# If hitting StartLimitBurst:
# "Start request repeated too quickly"
# Fix: Reset the failure counter
systemctl reset-failed myapp
# Or increase burst limit in unit:
[Unit]
StartLimitIntervalSec=60
StartLimitBurst=10
# ─────────────────────────────────────────────────────────────
# Issue: Service says "Active: active (running)" but actually dead
# (Process exited but systemd doesn't know)
# This means Type=simple with a daemon that backgrounds itself
# Fix: Use Type=forking with PIDFile=, or change to Type=notify
# with sd_notify() in the application
# ─────────────────────────────────────────────────────────────
# Issue: Dependency ordering problem
# "Service A fails because Service B isn't ready yet"
# Check the ordering
systemd-analyze critical-chain myapp.service
# Add explicit ordering:
[Unit]
After=postgresql.service
# AND ensure postgres signals readiness with Type=notify
# ─────────────────────────────────────────────────────────────
# Issue: Service can't bind to port 80 (privileged port)
# Root cause: service runs as non-root but port 80 < 1024
# Fix 1: Add capability
AmbientCapabilities=CAP_NET_BIND_SERVICE
# Fix 2: Use port 8080 and redirect with iptables
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
# Fix 3: Enable net.ipv4.ip_unprivileged_port_start (Linux 4.11+)
echo 80 > /proc/sys/net/ipv4/ip_unprivileged_port_start

Q1: What is the difference between Requires= and After= in a systemd unit?

Answer: Requires= defines a dependency relationship — if the required unit fails, this unit fails too. After= defines an ordering relationship — this unit starts after the listed units. They are independent. You typically need both: Requires=postgresql.service After=postgresql.service means “I need postgres, and I start after it starts.” Using only Requires= without After= means both units might start simultaneously (and ours might fail because postgres isn’t ready yet). Using only After= without Requires= means we start after postgres, but we don’t fail if postgres fails.

Q2: What is socket activation and why is it useful?

Answer: Socket activation means systemd holds open the listening socket and starts the service only when the first connection arrives. Benefits: (1) Faster boot — services don’t need to start until needed. (2) Zero-downtime restarts — while the service restarts, systemd buffers incoming connections in the kernel socket backlog. (3) Parallel startup — systemd can start dependent services immediately (they just write to the socket which is buffered) without waiting for the server to fully start. Used by D-Bus, SSH (in some distros), and many other services.

Q3: Explain the difference between Type=simple, Type=forking, and Type=notify.

Answer: Type=simple (default): systemd considers the service started as soon as ExecStart process begins. Doesn’t know when the service is actually ready to serve requests. Type=forking: Used when the service forks and the parent exits (traditional daemon pattern). systemd follows the child PID via PIDFile=. Problematic because systemd waits for the fork, not for actual readiness. Type=notify: The service calls sd_notify(READY=1) when it’s actually ready to serve. systemd only considers the service started after receiving this notification. Best for modern services — allows proper dependency ordering and eliminates race conditions.


FeaturePurposeKey Directive
Service unitsRun daemons/programsExecStart=, Type=
Timer unitsScheduled jobsOnCalendar=, Persistent=
Socket unitsOn-demand activationListenStream=
JournalStructured loggingjournalctl -u service
Drop-insOverride vendor unitssystemctl edit
HardeningSecurity restrictionsNoNewPrivileges=, PrivateTmp=

Chapter 2 (Boot Process) — understand the boot sequence and init system.


Advantages: Standardized across distros, parallel startup, built-in logging (journald), dependency management. Disadvantages: Monolithic and complex, steep learning curve compared to legacy SysVinit.


  • Using systemctl stop instead of systemctl disable for permanent removal — stop only halts now; disable prevents restart on next boot.
  • Writing ExecStart with absolute paths not quoted — paths with spaces break without quoting; always verify with systemd-analyze verify unit.service.
  • Forgetting Type=notify requires the app to call sd_notify(READY=1) — if the app doesn’t send ready notification, systemd thinks it failed.
  • Setting Restart=always without RestartSec — rapid restart loop can fork-bomb the system. Always set RestartSec=5.
  • Not using journalctl -u SERVICE --since today for per-service log filtering — looking at raw syslog is much slower.

SymptomCauseDiagnosisFix
Service fails to start (exit code 1)Misconfigured ExecStart or missing dependencyjournalctl -u myservice -n 50; systemctl status myservice -lFix ExecStart path; check After= dependencies; systemd-analyze verify myservice.service
Service starts but immediately exitsApplication exits 0 but systemd expects daemon modesystemctl status shows 'start request repeated too quickly'Set Type=simple (not forking) for non-daemon apps; or use Type=forking with PIDFile=
Timer unit not triggeringWrong OnCalendar syntax or timer not enabledsystemctl list-timers; systemctl status mytimer.timerVerify: systemd-analyze calendar ‘0/5:00’; enable+start the .timer unit, not the .service
journald consuming all disk spaceNo log rotation configured or rate limiting disabledjournalctl --disk-usage; du -sh /var/log/journalSet SystemMaxUse=1G in /etc/systemd/journald.conf; systemctl restart systemd-journald

Objective: Write and manage a production-quality systemd service.

Terminal window
# 1. Create a simple service unit
cat > /tmp/myapp.service << 'EOF'
[Unit]
Description=My Application
After=network.target
Wants=network.target
[Service]
Type=simple
User=nobody
ExecStart=/bin/sleep 300
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
EOF
sudo cp /tmp/myapp.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start myapp
sudo systemctl status myapp
# 2. View its logs
journalctl -u myapp -f
# 3. Validate the unit file
systemd-analyze verify /etc/systemd/system/myapp.service
# 4. Create a timer for it
cat > /tmp/myapp.timer << 'EOF'
[Unit]
Description=Run myapp every 5 minutes
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.target
EOF
# 5. Check boot performance
systemd-analyze critical-chain

  1. Write a systemd service that restarts nginx if it crashes, waits 10s before restart, and logs to the journal. Include resource limits (MemoryMax=256M).
  2. Create a systemd timer that runs a backup script every day at 2 AM. Verify with systemd-analyze calendar '02:00:00'.
  3. Use a drop-in override to increase the restart limit for an existing service: systemctl edit nginx and set StartLimitBurst=10.
  4. Write a service with Type=notify — implement the sd_notify('READY=1') call in a simple Python script using the systemd module.
  5. Use journalctl to find all services that have failed in the last 7 days: journalctl -p err --since '7 days ago'.

  • systemd unit types: service, timer, socket, target, mount, device, slice — each has a specific purpose.
  • Service Type=: simple (default), forking (old-style daemons), notify (sends sd_notify), oneshot (runs once).
  • After= = dependency ordering; Wants= = soft dependency (failure OK); Requires= = hard dependency (fails together).
  • Timers replace cron — OnCalendar= uses flexible syntax; systemctl list-timers shows next trigger.
  • journald is structured: journalctl -u SERVICE -S today -p err — filter by unit, time, priority.
  • Drop-in overrides (systemctl edit) survive package upgrades — always prefer over editing unit files directly.

  • systemd man pages: man 5 systemd.service, man 5 systemd.timer
  • systemd docs: https://systemd.io/
  • Linux Service Management Made Easy with systemd — free online resource
  • journald: man 1 journalctl — complete filtering reference


Last Updated: July 2026