Systemd Deep Dive — Services, Timers, and Boot Targets

Your app crashes at 3 AM and nobody restarts it — systemd can fix that forever. Systemd is the init system and service manager on virtually every modern Linux distribution, and understanding it is non-negotiable for DevOps work.

Systemd Basics — systemctl

systemctl is your primary interface to systemd. Here are the commands you'll use daily.

# Start, stop, restart a service
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx

# Reload config without restarting (zero-downtime for supported services)
sudo systemctl reload nginx

# Enable service to start at boot / disable it
sudo systemctl enable nginx
sudo systemctl disable nginx

# Check service status (shows logs, PID, memory usage)
systemctl status nginx

The difference between enable and start trips up beginners:

Command	What It Does	When
start	Starts the service right now	Immediately
enable	Creates symlinks so it starts at boot	Next boot
enable --now	Enables AND starts immediately	Both

# The shortcut — enable and start in one command
sudo systemctl enable --now nginx

Listing and Inspecting Services

# List all running services
systemctl list-units --type=service --state=running

# List all services (including inactive)
systemctl list-units --type=service --all

# List failed services — your first check after a reboot
systemctl --failed

# Show the full service unit file
systemctl cat nginx.service

# Show all properties of a service
systemctl show nginx.service

# Check if a specific service is active/enabled
systemctl is-active nginx
systemctl is-enabled nginx

Creating a Custom Service

This is where systemd becomes powerful. Let's say you have a Node.js API that needs to run as a daemon.

# Create a service unit file
sudo tee /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=My Node.js API Server
Documentation=https://github.com/myorg/myapp
After=network.target
Wants=network.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp/data /var/log/myapp

# Environment
Environment=NODE_ENV=production
Environment=PORT=3000
EnvironmentFile=-/opt/myapp/.env

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd to pick up the new file
sudo systemctl daemon-reload

# Enable and start
sudo systemctl enable --now myapp.service

# Check it's running
systemctl status myapp.service

Key directives explained:

Directive	Value	Purpose
Restart	on-failure	Restart only on non-zero exit codes
Restart	always	Restart no matter what (good for daemons)
RestartSec	5	Wait 5 seconds before restarting
StartLimitBurst	5	Max 5 restarts...
StartLimitIntervalSec	60	...within 60 seconds, then stop trying

Systemd Timers — The Modern Cron

Systemd timers are more powerful than cron: they support dependencies, randomized delays, persistent scheduling (runs missed jobs after boot), and proper logging through journald.

You need two files — a timer unit and a matching service unit.

# Create the service that does the actual work
sudo tee /etc/systemd/system/db-backup.service << 'EOF'
[Unit]
Description=Database Backup Script

[Service]
Type=oneshot
User=backup
ExecStart=/opt/scripts/backup-db.sh
StandardOutput=journal
EOF

# Create the timer that triggers the service
sudo tee /etc/systemd/system/db-backup.timer << 'EOF'
[Unit]
Description=Run database backup daily at 2 AM

[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable and start the timer (not the service!)
sudo systemctl daemon-reload
sudo systemctl enable --now db-backup.timer

# List all active timers with next run time
systemctl list-timers --all

# Manually trigger the backup right now (for testing)
sudo systemctl start db-backup.service

# Check the last run result
systemctl status db-backup.service
journalctl -u db-backup.service --since today

Timer schedule examples:

OnCalendar Value	Meaning
*-*-* 02:00:00	Every day at 2 AM
Mon *-*-* 09:00:00	Every Monday at 9 AM
*-*-01 00:00:00	First day of every month
*-*-* *:00/15:00	Every 15 minutes
weekly	Every Monday at midnight

Boot Targets

Targets are systemd's replacement for SysVinit runlevels. They group services together.

# See current default target
systemctl get-default

# Set default target (e.g., no GUI for servers)
sudo systemctl set-default multi-user.target

# Available targets
# graphical.target  — Full GUI (runlevel 5)
# multi-user.target — Multi-user CLI (runlevel 3)
# rescue.target     — Single user mode (runlevel 1)
# emergency.target  — Minimal shell, no mounts

# Switch target without rebooting (careful!)
sudo systemctl isolate multi-user.target

Journalctl — Reading Systemd Logs

Every service managed by systemd logs to the journal. journalctl is how you read it.

# View logs for a specific service
journalctl -u nginx.service

# Follow logs in real-time (like tail -f)
journalctl -u myapp.service -f

# Logs since last boot
journalctl -b

# Logs from the previous boot (useful after crash)
journalctl -b -1

# Logs from the last hour
journalctl --since "1 hour ago"

# Logs between two timestamps
journalctl --since "2025-05-24 08:00" --until "2025-05-24 12:00"

# Show only errors and above
journalctl -u nginx.service -p err

# Output as JSON (pipe to jq for parsing)
journalctl -u nginx.service -o json-pretty --no-pager | head -50

Priority levels for the -p flag:

Level	Name	Use
0	emerg	System is unusable
1	alert	Immediate action required
2	crit	Critical conditions
3	err	Error conditions
4	warning	Warning conditions
5	notice	Normal but significant
6	info	Informational
7	debug	Debug-level messages

Service Dependencies and Ordering

Control the order services start and their relationships.

# View the dependency tree for a service
systemctl list-dependencies nginx.service

# In your unit file, use these directives:
# After=  — Start after these units (ordering)
# Before= — Start before these units
# Requires= — Hard dependency (if dependency fails, this fails too)
# Wants= — Soft dependency (if dependency fails, continue anyway)
# BindsTo= — Strongest dependency (stops if dependency stops)

Example: a web app that needs the database running first.

sudo tee /etc/systemd/system/webapp.service << 'EOF'
[Unit]
Description=Web Application
After=network.target postgresql.service redis.service
Requires=postgresql.service
Wants=redis.service

[Service]
Type=simple
ExecStart=/opt/webapp/start.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

This means: start after PostgreSQL and Redis, require PostgreSQL (fail if it's not running), want Redis (start without it if necessary).

Troubleshooting Failed Services

When things go wrong, here's your debugging checklist.

# Step 1: Check status and recent logs
systemctl status myapp.service

# Step 2: Get the full log output
journalctl -u myapp.service --no-pager -n 100

# Step 3: Verify the unit file syntax
systemd-analyze verify /etc/systemd/system/myapp.service

# Step 4: Check for resource limits
systemctl show myapp.service | grep -i limit

# Step 5: Analyze boot time and slow services
systemd-analyze blame
systemd-analyze critical-chain myapp.service

# Step 6: If stuck in "activating", check the process
systemctl show myapp.service -p MainPID

---

Next in our Linux series: Linux Log Analysis — learn how to find problems in logs before they become outages, using journalctl, grep patterns, awk, and log rotation strategies.