Systemd Units and Services

Chapter 18: systemd Units and Services

Creating and Managing Custom systemd Services

Why This Matters in DevOps/SRE

systemd units are how you deploy and manage applications in Linux:

                 systemd Units for DevOps/SRE
+------------------------------------------------------------------+
|                                                                   |
|    Application Deployment:                                           |
|    +----------------------------------------------------------+  |
|    | All applications run as systemd services                |  |
|    | Docker/Kubernetes use systemd under the hood           |  |
|    | Custom apps need unit files for production            |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Automation & Reliability:                                          |
|    +----------------------------------------------------------+  |
|    | Auto-restart on failure → Resilience                 |  |
|    | Dependencies → Proper startup ordering               |  |
|    | Resource limits → Prevent resource exhaustion        |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Monitoring & Observability:                                       |
|    +----------------------------------------------------------+  |
|    | systemd-analyze → Identify slow startup              |  |
|    | journalctl → Centralized logging                     |  |
|    | systemctl status → Quick health checks               |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+

Practical Impact:

• Deploy custom applications reliably
• Set up proper logging and monitoring
• Configure auto-restart and dependencies
• Manage resources with limits

---

18.1 Understanding Service Unit Files

Unit File Syntax

Service unit files are the configuration files that tell systemd how to manage a service. They follow the INI format with sections for different aspects of the service.

                 Service Unit File Structure
+------------------------------------------------------------------+
|                                                                   |
|    Location: /etc/systemd/system/<name>.service                  |
|                                                                   |
|    Format:                                                        |
|    +----------------------------------------------------------+  |
|    | [Unit]                                                    |  |
|    | Description=Human readable description                    |  |
|    | Documentation=URL or man page                            |  |
|    | After=network.target syslog.target                        |  |
|    | Before=shutdown |  |
|.target                                       | Wants=network.target                                     |  |
|    | Requires=network.target                                  |  |
|    |                                                           |  |
|    | [Service]                                                 |  |
|    | Type=simple                                               |  |
|    | ExecStart=/usr/bin/myapp --config /etc/myapp.conf       |  |
|    | ExecStop=/usr/bin/myapp --stop                          |  |
|    | ExecReload=/bin/kill -HUP $MAINPID                       |  |
|    | Restart=on-failure                                        |  |
|    | RestartSec=5                                              |  |
|    | User=myappuser                                           |  |
|    | Group=myappgroup                                          |  |
|    | WorkingDirectory=/var/lib/myapp                          |  |
|    | Environment=NODE_ENV=production                          |  |
|    |                                                           |  |
|    | [Install]                                                 |  |
|    | WantedBy=multi-user.target                               |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+

Example: Custom Application Service

[Unit]
Description=My Custom Application Service
Documentation=https://myapp.example.com/docs
After=network.target postgresql.service
Wants=network.target
After=syslog.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/var/lib/myapp
Environment="NODE_ENV=production"
Environment="PORT=3000"
EnvironmentFile=-/etc/myapp/env

ExecStart=/usr/bin/node /opt/myapp/server.js
ExecStop=/bin/kill -TERM $MAINPID
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=10
TimeoutStartSec=30
TimeoutStopSec=30

StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

---

18.2 Service Type Deep Dive

Type: simple (Most Common)

[Service]
Type=simple
ExecStart=/usr/bin/myapp

# Main process is the ExecStart process
# If it exits, service is considered stopped
# systemd considers it "active" while process runs

# Use when:
# - Your application runs in the foreground
# - No forking required
# - Most common for modern applications

Type: forking (Traditional Daemons)

[Service]
Type=forking
ExecStart=/usr/bin/mydemon --daemon
PIDFile=/run/mydemon.pid

# Parent process forks child
# systemd waits for parent to exit
# Uses PIDFile to track main process
# Service is "active" while child runs

# Use when:
# - Traditional daemon that forks to background
# - Parent exits after forking
# - Legacy applications

Type: oneshot (One-Time Tasks)

[Service]
Type=oneshot
ExecStart=/usr/bin/setup-script.sh
RemainAfterExit=yes

# Runs once and exits
# RemainAfterExit=yes keeps it "active" after completion
# Useful for initialization tasks
# Exit code determines success/failure

# Use when:
# - One-time setup/configuration
# - Initialization scripts
# - Services that don't keep running

Type: notify (Ready Notifications)

[Service]
Type=notify
ExecStart=/usr/bin/myapp

# systemd waits for READY notification
# Service sends sd_notify("READY=1") when ready
# More reliable than Type=simple for services needing init time
# Requires application support for notifications

# Use when:
# - Application needs time to initialize
# - Application can send notifications
# - Better than simple when startup time varies

Type: dbus (D-Bus Based)

[Service]
Type=dbus
BusName=com.example.MyService
ExecStart=/usr/bin/myapp

# Waits for D-Bus name to appear
# Service marked active when BusName is acquired
# Common for system services using D-Bus

# Use when:
# - Service registers with D-Bus
# - Uses D-Bus for communication
# - System services

---

18.3 Essential Service Directives

Execution Control

[Service]
# Start command (required)
ExecStart=/path/to/command arg1 arg2

# Pre-start commands (run before ExecStart)
ExecStartPre=/path/to/script.sh
ExecStartPre=/bin/mkdir -p /var/run/myapp

# Post-start commands (run after ExecStart starts)
ExecStartPost=/path/to/post-start.sh

# Stop command (alternative to signal)
ExecStop=/path/to/stop-script.sh

# Post-stop commands (run after ExecStop)
ExecStopPost=/path/to/cleanup.sh

# Reload command (for config changes)
ExecReload=/bin/kill -HUP $MAINPID

# Note: $MAINPID is the PID of the main process

Restart Behavior

[Service]
# When to restart
# on-failure (default) - non-zero exit, timeout, crashed
# on-success - clean exit (for oneshot)
# always - always restart
# on-abnormal - timeout, crash, failed signal
Restart=on-failure

# Time between stop and start
RestartSec=5

# Maximum restart attempts
StartLimitIntervalSec=300   # Within this time window
StartLimitBurst=5          # This many restarts allowed

# What to do if limit reached
StartLimitAction=reboot
# Options: none, reboot, reboot-force, reboot-immediate

Environment and Working Directory

[Service]
# Single environment variable
Environment="VAR1=value1"
Environment="VAR2=value2"

# Multiple variables
Environment="VAR1=value1" "VAR2=value2"

# From file (lines like VAR=value)
EnvironmentFile=/etc/myapp/env

# Working directory
WorkingDirectory=/var/lib/myapp

# User and group to run as
User=myappuser
Group=myappgroup

Output Handling

[Service]
# Where to send stdout/stderr
# journal, syslog, kmsg, console, null

# Combined output to journal
StandardOutput=journal
StandardError=journal

# Separate outputs
StandardOutput=file:/var/log/myapp/stdout.log
StandardError=file:/var/log/myapp/stderr.log

# Syslog identification
SyslogIdentifier=myapp
SyslogFacility=user
SyslogLevel=info
SyslogLevelPrefix=yes

---

18.4 Security Hardening

Protected Execution

[Service]
# Isolate from system
ProtectSystem=strict         # Read-only /usr, /boot, /etc
ProtectSystem=full          # + /run, /var (except some dirs)
ProtectSystem=no           # Disabled (default before 247)

# Protect home directory
ProtectHome=yes             # Hide /home, /root, /run/user
ProtectHome=read-only       # Read-only access
ProtectHome=no              # No protection

# Additional protections
ProtectKernelTunables=yes   # Read-only /proc/sys
ProtectKernelModules=yes    # Block module loading
ProtectControlGroups=yes   # Read-only cgroups

# New privileges
NoNewPrivileges=true        # Prevent privilege escalation

# Private /tmp
PrivateTmp=yes              # Private /tmp

# Network access
PrivateNetwork=yes         # No network access
ProtectNetwork=yes         # Virtual network only

Seccomp and Capabilities

[Service]
# System call filtering
SystemCallFilter=@system-service
SystemCallFilter=~@clock @debug @ipc @module @mount @obsolete @privileged @reboot

# System call error number
SystemCallErrno=EPERM

# Capability management
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_CHOWN

# Memory protection
MemoryDenyWriteExecute=true
RestrictRealtime=true
RestrictNamespaces=yes

Filesystem Restrictions

[Service]
# Read-only filesystem (except specified paths)
ReadWritePaths=/var/lib/myapp
ReadWritePaths=/var/log/myapp
ReadWritePaths=/run/myapp

# Temporary filesystem
TemporaryFileSystem=/tmp:size=10M

# Bind mounts
BindReadOnlyPaths=/etc/ssl/certs:/etc/ssl/certs:ro

---

18.5 Resource Limits

Memory Limits

[Service]
# Hard limit - process cannot exceed
MemoryMax=1G

# Soft limit - process gets priority up to this
MemoryHigh=512M

# Swap limit
MemorySwapMax=100M

CPU Limits

[Service]
# CPU quota (percentage of one CPU)
CPUQuota=50%       # Max 50% of one CPU
CPUQuota=200%      # Max 2 CPUs

# CPU affinity (specific cores)
CPUAffinity=0      # Core 0 only
CPUAffinity=0-3   # Cores 0-3
CPUAffinity=0,2,4 # Cores 0, 2, 4

I/O Limits

[Service]
# Bandwidth limits
IOReadBandwidthMax=/var/lib/mysql 1G
IOWriteBandwidthMax=/var/lib/mysql 1G

# IOPS limits
IOPSReadMax=10000
IOPSWriteMax=5000

Process Limits

[Service]
# Max processes/threads
TasksMax=100

# File descriptors
LimitNOFILE=1000

# Other limits
LimitNPROC=50
LimitAS=2G
LimitCORE=0       # Disable core dumps

---

18.6 Socket Activation

Socket Unit

[Unit]
Description=My Application Socket
PartOf=myapp.service

[Socket]
ListenStream=/run/myapp.sock
SocketMode=0660
SocketUser=myapp
SocketGroup=myapp

# Or TCP socket
ListenStream=8080
BindIPv6Only=both

# Socket service activation
Service=myapp.service

Service for Socket Activation

[Service]
Type=notify
ExecStart=/usr/bin/myapp
SocketBindToLowerLevel=true

# Note: Remove any Port= or similar directives
# Service receives socket as fd

Creating Socket-Activated Service

# Create socket unit
sudo tee /etc/systemd/system/myapp.socket << 'EOF'
[Unit]
Description=My App Socket
PartOf=myapp.service

[Socket]
ListenStream=9000
Accept=false

[Install]
WantedBy=sockets.target
EOF

# Create service unit (without traditional port binding)
sudo tee /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=My Application
After=network.target
Requires=myapp.socket

[Service]
Type=notify
ExecStart=/usr/bin/myapp
User=myapp

[Install]
WantedBy=multi-user.target
EOF

# Reload and enable
sudo systemctl daemon-reload
sudo systemctl enable --now myapp.socket

---

18.7 Timer Units for Scheduling

Timer Unit File

[Unit]
Description=Daily Database Backup
Requires=backup.service

[Timer]
# Calendar-based scheduling
OnCalendar=daily
OnCalendar=Mon *-*-* 02:00:00    # Every Monday at 2 AM
OnCalendar=*-*-* 12:00:00        # Every day at noon
OnCalendar=*:0/15                # Every 15 minutes
OnCalendar=hourly                 # Every hour

# Time from events
OnActiveSec=1h                    # 1 hour after activation
OnBootSec=30min                   # 30 minutes after boot
OnUnitActiveSec=1h               # 1 hour after last run
OnUnitInactiveSec=1h             # 1 hour after last ended

# Options
Persistent=true                   # Run missed jobs on boot
RandomizedDelaySec=1h            # Random delay to prevent storms
WakeSystem=true                  # Wake from suspend
AccuracySec=1us                  # Accuracy (default 1min)

[Install]
WantedBy=timers.target

Service for Timer

[Unit]
Description=Daily Database Backup

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup.sh

Managing Timers

# Enable and start
sudo systemctl enable --now backup.timer

# List timers
systemctl list-timers --all
systemctl list-timers --all | grep backup

# Manual trigger (for testing)
sudo systemctl start backup.service

# View next run time
systemctl list-timers --all | grep backup

# Timer status
systemctl status backup.timer

# View timer details
systemctl cat backup.timer

# View missed triggers
journalctl -u backup.timer -u backup.service

---

18.8 Path Units for File Monitoring

Path Unit File

[Unit]
Description=Watch directory for new files
PartOf=processor.service

[Path]
# Monitor file/directory
DirectoryNotEmpty=/var/spool/myapp
PathExists=/tmp/trigger
PathChanged=/etc/myapp/config.conf

# Options
Unit=processor.service

[Install]
WantedBy=multi-user.target

Service for Path Unit

[Unit]
Description=Process new files

[Service]
Type=oneshot
ExecStart=/usr/local/bin/process-files.sh

Using Path Units

# Create path unit
sudo tee /etc/systemd/system/watchdir.path << 'EOF'
[Unit]
Description=Watch /var/spool/incoming
[Path]
DirectoryNotEmpty=/var/spool/incoming
Unit=process-files.service

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl enable --now watchdir.path

# Check status
systemctl status watchdir.path

---

18.9 Drop-in Units

Understanding Drop-ins

Drop-in files allow you to modify unit files without editing the original vendor files. They are stored in a subdirectory with .d suffix.

                 Drop-in Directory Structure
+------------------------------------------------------------------+
|                                                                   |
|    Main unit file:                                                |
|    /usr/lib/systemd/system/nginx.service                        |
|                                                                   |
|    Drop-in directory:                                             |
|    /etc/systemd/system/nginx.service.d/                         |
|                                                                   |
|    Drop-in files (override specific directives):                 |
|    /etc/systemd/system/nginx.service.d/override.conf           |
|    /etc/systemd/system/nginx.service.d/memory.conf             |
|                                                                   |
|    Drop-in files can add new directives or override existing    |
|                                                                   |
+------------------------------------------------------------------+

Creating Drop-ins

# Method 1: Using systemctl edit (recommended)
sudo systemctl edit nginx

# This opens editor with blank file
# Add overrides, save and exit

# Method 2: Manual drop-in file
sudo mkdir -p /etc/systemd/system/nginx.service.d
sudo tee /etc/systemd/system/nginx.service.d/override.conf << 'EOF'
[Service]
Restart=always
MemoryMax=512M
Environment="NGINX_PORT=8080"
EOF

# Method 3: Full edit
sudo systemctl edit --full nginx

# View effective unit
systemctl show nginx

# View drop-ins
systemctl cat nginx

Common Override Examples

# Override restart policy
sudo systemctl edit nginx
[Service]
Restart=always

# Override memory limit
sudo systemctl edit nginx
[Service]
MemoryMax=1G

# Add environment variables
sudo systemctl edit nginx
[Service]
Environment="NODE_ENV=production"
Environment="PORT=3000"

# Add extra mounts
sudo systemctl edit nginx
[Service]
ReadWritePaths=/data

# Override user
sudo systemctl edit nginx
[Service]
User=nginx
Group=nginx

---

18.10 Troubleshooting Service Issues

Common Issues and Solutions

# Service won't start
# 1. Check status
sudo systemctl status myapp

# 2. Check logs
sudo journalctl -u myapp -n 50
sudo journalctl -u myapp --since "10 minutes ago"
sudo journalctl -u myapp -xe

# 3. Verify unit file syntax
sudo systemd-analyze verify myapp.service

# 4. Check dependencies
sudo systemctl list-dependencies myapp
sudo systemd-analyze dot myapp.service | dot -Tsvg > deps.svg

# 5. Check for conflicts
sudo systemctl list-units --state=failed
sudo systemctl failed

# 6. Debug starting
sudo systemd-analyze service-order
sudo strace -f systemctl start myapp

# Permission issues
# Check file permissions
ls -la /etc/systemd/system/myapp.service
ls -la /usr/bin/myapp

# Check user exists
id myappuser

# Path issues
# Verify ExecStart path exists
which myapp
ls -la /path/to/myapp

Viewing Detailed Service Information

# Full unit properties
systemctl show nginx

# Specific properties
systemctl show nginx -p MainPID
systemctl show nginx -p MemoryCurrent
systemctl show nginx -p CPUUsageNSec

# Unit file with defaults
systemctl cat nginx

# Unit file dependencies
systemctl list-dependencies nginx
systemctl list-dependencies --reverse nginx

# Resource usage
systemctl status nginx
systemctl show nginx | grep -i memory

# Failed reason
systemctl failure nginx

Testing Services

# Check if service can start
sudo systemctl start myapp
echo $?  # 0 = success

# Test start without systemd
sudo -u myappuser /usr/bin/myapp

# Run in foreground for debugging
sudo systemctl stop myapp
/usr/bin/myapp --debug

# Check environment
sudo systemd-run --scope -p User=myappuser env | grep -i myapp

# Temporary test service
sudo systemd-run --scope /bin/bash

---

18.11 Best Practices for Service Units

                 Service Unit Best Practices
+------------------------------------------------------------------+
|                                                                   |
|    Always Use:                                                    |
|    +----------------------------------------------------------+  |
|    | ✓ Description= for all services                         |  |
|    | ✓ After= with Wants= for dependencies                   |  |
|    | ✓ Restart= with RestartSec= for critical services      |  |
|    | ✓ User= and Group= (not running as root)               |  |
|    | ✓ WorkingDirectory= for relative paths                 |  |
|    | ✓ StandardOutput=journal for logging                    |  |
|    | ✓ ProtectSystem=yes for security                        |  |
|    | ✓ PrivateTmp=yes                                        |  |
|    | ✓ NoNewPrivileges=yes                                  |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Avoid:                                                          |
|    +----------------------------------------------------------+  |
|    | ✗ Running as root unless absolutely necessary          |  |
|    | ✗ Complex shell commands in ExecStart                 |  |
|    | ✗ Ignoring exit codes                                   |  |
|    | ✗ Missing dependencies                                  |  |
|    | ✗ Hardcoded paths without verification                |  |
|    | ✗ Using deprecated directives                          |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Testing:                                                        |
|    +----------------------------------------------------------+  |
|    | ✓ Always test with systemctl start first              |  |
|    | ✓ Check journalctl output                              |  |
|    | ✓ Verify with systemd-analyze verify                  |  |
|    | ✓ Test after reboot                                    |  |
|    | ✓ Monitor with systemctl status in production         |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+

---

18.12 Interview Questions

Basic Questions

1. What is a systemd unit file?

   • Configuration file that defines a resource or service managed by systemd

2. What are the main sections in a service unit file?

   • [Unit], [Service], [Install]

3. What does Type=simple mean?

   • Main process is the ExecStart process; service is active while it runs

4. How do you make a service start at boot?

   • systemctl enable servicename
   • WantedBy= in [Install] section

5. What is the difference between ExecStart and ExecStartPre?

   • ExecStart is the main command; ExecStartPre runs before it

Intermediate Questions

6. What does Restart=on-failure do?

   • Automatically restarts the service if it exits with non-zero status

7. How do you override settings from a vendor unit file?

   • Using drop-in files in /etc/systemd/system/.service.d/

8. What is socket activation?

   • Service starts on-demand when a connection is made to its socket

9. How do you set environment variables in a service?

   • Environment= directive or EnvironmentFile=

10. What is the purpose of $MAINPID in systemd units?

    • References the PID of the main process started by ExecStart

Advanced Questions

11. Explain the difference between Requires and Wants

    • Requires is a hard dependency (fails if dependency fails)
    • Wants is a soft dependency (continues if dependency fails)

12. How do you debug a failing service?

    • systemctl status, journalctl -u, systemd-analyze verify, check dependencies

13. What security measures should you implement in service units?

    • ProtectSystem, PrivateTmp, NoNewPrivileges, User/Group, read-only paths

14. How do timer units work?

    • Schedule service execution based on calendar or time intervals
    • Alternative to cron for systemd-managed scheduling

15. What is the purpose of RemainAfterExit=yes?

    • Keeps the service marked as “active” even after the process exits (for oneshot)

---

Common Mistakes & Anti-Patterns

1. Running as Root

# ❌ WRONG: Running services as root
[Service]
User=root
# Any vulnerability = full system compromise

# ✅ CORRECT: Use dedicated service user
[Service]
User=myapp
Group=myapp
# Create user: useradd -r -s /sbin/nologin myapp

2. Not Using Environment Files

# ❌ WRONG: Hardcoding credentials in unit file
[Service]
Environment="PASSWORD=secret123"
# Exposed in process list!

# ✅ CORRECT: Use EnvironmentFile
[Service]
EnvironmentFile=/etc/myapp/env
# /etc/myapp/env contains:
# PASSWORD=secret123
# DATABASE_URL=postgres://...

3. Missing Restart Logic

# ❌ WRONG: No restart policy
[Service]
Type=simple
# Just runs once, doesn't restart on crash

# ✅ CORRECT: Configure restart behavior
[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=3

4. Not Setting Working Directory

# ❌ WRONG: Not setting working directory
[Service]
ExecStart=/usr/bin/myapp
# Uses / as working directory
# Relative paths fail!

# ✅ CORRECT: Set working directory
[Service]
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/myapp

---

Summary

Creating systemd service units requires understanding:

                 Quick Reference
+------------------------------------------------------------------+
|                                                                   |
|    Essential Directives:                                          |
|    +----------------------------------------------------------+  |
|    | Type=           | Service startup behavior              |  |
|    | ExecStart=      | Command to start service              |  |
|    | Restart=        | When to restart                       |  |
|    | User/Group=     | Run as non-root                       |  |
|    | After=          | Ordering dependencies                  |  |
|    | WantedBy=       | Boot target                            |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Service Types:                                                 |
|    +----------------------------------------------------------+  |
|    | simple     | Default, main process is service          |  |
|    | forking   | Parent forks, uses PIDFile                 |  |
|    | oneshot   | Runs once, use RemainAfterExit             |  |
|    | notify    | Waits for READY notification               |  |
|    | dbus      | Waits for D-Bus name                       |  |
|    +----------------------------------------------------------+  |
|                                                                   |
|    Key Commands:                                                  |
|    +----------------------------------------------------------+  |
|    | systemctl daemon-reload        | Reload unit files       |  |
|    | systemctl edit <service>      | Create drop-in          |  |
|    | systemctl cat <service>       | View unit file          |  |
|    | journalctl -u <service>       | View logs                |  |
|    | systemd-analyze verify        | Check syntax            |  |
|    +----------------------------------------------------------+  |
|                                                                   |
+------------------------------------------------------------------+