Service Management with systemd: Units, Logs, and Auto-Restart
You will manage existing services and author a custom systemd unit that restarts automatically on failure. By the end you will have a resilient daemon with logs in journald and a clean enable-on-boot configuration.
Learning Objectives
- Start, stop, enable, and inspect services with systemctl.
- Write a custom unit with automatic restart.
- Read service logs with journalctl.
- Time: ~2 hours · Difficulty: Intermediate · Prereqs: a systemd-based Linux server.
Architecture Overview
Environment Setup
You will need: a Linux server with systemd and a simple program or script to run as a service.
Before you begin: confirm systemd is the init system with systemctl -version.
Prerequisites
- A systemd-based Linux distribution (any modern Debian, Ubuntu, Fedora, RHEL).
- Root or sudo access to install unit files.
- A test binary or script to run as a service (/usr/local/bin/app used here).
- Basic comfort with the terminal and text editors.
Step-by-Step Execution
Understanding state and dependencies is the basis of all service management.
systemctl status ssh --no-pager && systemctl list-dependencies ssh
Expected output:
ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
Active: active (running)What this teaches: status shows state, main PID, and recent logs; list-dependencies reveals ordering, which explains why a service waits at boot. Note: the unit is named ssh.service on Debian/Ubuntu but sshd.service on RHEL/Fedora; adjust the command to your distro.
The Restart directive makes the daemon self-heal after a crash, key for reliability.
printf '[Unit]\nDescription=My App\n[Service]\nExecStart=/usr/local/bin/app\nRestart=on-failure\nRestartSec=3\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/app.serviceExpected output:
[Unit]
Description=My App
[Service]
ExecStart=/usr/local/bin/app
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target
(unit file written)What this teaches: A minimal unit needs [Unit], [Service] with ExecStart, and [Install]; Restart=on-failure plus RestartSec makes the daemon self-healing. Type defaults to simple, so systemd tracks the foreground process you start; use Type=forking only for daemons that background themselves.
sudo systemctl daemon-reload && sudo systemctl enable --now app.service
Expected output:
(no output on success; verify with: systemctl is-active app.service)
activeWhat this teaches: daemon-reload makes systemd re-read unit files; enable wires it into the boot target; --now starts it immediately too.
Expected output:
What this teaches: journalctl -u filters the journal to one unit's logs; stdout/stderr of the service land here automatically with no logfile config.
How It Works
systemd is PID 1 and the service manager: it parses unit files into an in-memory dependency graph, starts units in topological order, and supervises them afterwards. A service unit declares what to run (ExecStart), when (After=, WantedBy=), and how failures are handled (Restart=). This declarative model replaces fragile init scripts with state systemd can reason about.
Restart policies are the reliability core. on-failure restarts on crashes but not clean exits; always restarts regardless; RestartSec paces attempts to avoid hot loops. Combined with the journal capturing all output, a well-written unit is observable and self-healing without extra tooling.
Enablement is separate from starting: enable creates symlinks in .wants directories so the unit starts at boot, while start affects only now. systemctl is-enabled versus is-active answers the two different questions of 'will it start' and 'is it running'.
Progress So Far
Testing & Validation
sudo kill -9 $(systemctl show -p MainPID --value app.service); sleep 4; systemctl is-active app.serviceAfter killing the process you should see active again within seconds, proving auto-restart works.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Unit not found after creating the file | daemon-reload was skipped | Run systemctl daemon-reload after every unit file change |
| Service fails with exit 203 | ExecStart path wrong or not executable | Use absolute paths and chmod +x the binary |
| Restart loop spams the journal | Restart=always with an instantly-crashing ExecStart | Add RestartSec=5 and StartLimitIntervalSec/StartLimitBurst to pace attempts |
| Enabled but not starting at boot | WantedBy target wrong or enable created no symlink | Check systemctl is-enabled and that [Install] names multi-user.target |
| Environment variables missing | Units do not inherit your shell environment | Set them with Environment= lines or EnvironmentFile= |
| journalctl -u shows nothing | StandardOutput redirected or syslog forwarding only | Keep default journal capture; check with systemctl status for recent lines |
Troubleshooting
- Unit not found: run
systemctl daemon-reloadafter creating the file. - Service flaps: add
StartLimitIntervalSec/StartLimitBurstto avoid restart loops. - No logs: ensure the program writes to stdout/stderr, which journald captures.
Extension Ideas
- Pair the service with a systemd timer for scheduled runs.
- Sandbox it with directives like
ProtectSystem=strict. - Forward journald to a central log host (see Log Management).
Key Results
- Authored a custom unit that auto-restarts within ~3 seconds of failure.
- Enabled the service to start on boot reliably.
- Captured all service output in journald.
- Proved self-healing with a kill-and-recover test.
Key Takeaways
- Units declare what, when, and how-failures-handled; systemd supervises.
- daemon-reload after every unit edit; enable for boot, start for now.
- Restart=on-failure plus RestartSec gives self-healing services.
- journalctl -u is the log view; no logfile plumbing needed.
- Absolute ExecStart paths and Environment= lines prevent classic failures.
Practice Challenges
Challenge 1: Self-healing app
Goal: write a unit for a script that crashes randomly, prove with journalctl that systemd restarted it each time, and count restarts.
Hint: script exits 1 on a coin flip; watch systemctl show -p NRestarts.
Challenge 2: Dependency ordering
Goal: make app.service start only after network-online.target and a postgres.service, then verify the order in the boot graph.
Hint: After= plus Wants=, verified with systemd-analyze critical-chain.