Service Management with systemd: Units, Logs, and Auto-Restart
You will manage existing services and author a custom systemd unit that restarts automatically on failure. By the end you will have a resilient daemon with logs in journald and a clean enable-on-boot configuration.
Learning Objectives
- Start, stop, enable, and inspect services with systemctl.
- Write a custom unit with automatic restart.
- Read service logs with journalctl.
- Time: ~2 hours · Difficulty: Intermediate · Prereqs: a systemd-based Linux server.
Architecture Overview
Environment Setup
You will need: a Linux server with systemd and a simple program or script to run as a service.
Before you begin: confirm systemd is the init system with systemctl -version.
Prerequisites
- A systemd-based Linux distribution (any modern Debian, Ubuntu, Fedora, RHEL).
- Root or sudo access to install unit files.
- A test binary or script to run as a service (/usr/local/bin/app used here).
- Basic comfort with the terminal and text editors.
Step-by-Step Execution
The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run.
Understanding state and dependencies is the basis of all service management.
systemctl status ssh --no-pager && systemctl list-dependencies sshOutput (first 14 lines):
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
Active: active (running) since Sat 2026-11-14 17:32:04 EST; 14ms ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 457 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 458 (sshd)
Tasks: 1 (limit: 7040)
Memory: 1.1M (peak: 1.5M)
CPU: 14ms
CGroup: /docker/e18ffc53785729c64117a4ec9895a6a9cedb99e3dbbf9d7b31d42a665b4cd81f/system.slice/ssh.service
└─458 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
ssh.service
What this teaches: status shows state, main PID, and recent logs; list-dependencies reveals ordering, which explains why a service waits at boot. Note: the unit is named ssh.service on Debian/Ubuntu but sshd.service on RHEL/Fedora; adjust the command to your distro.
The Restart directive makes the daemon self-heal after a crash, key for reliability.
printf '[Unit]\nDescription=My App\n[Service]\nExecStart=/usr/local/bin/app\nRestart=on-failure\nRestartSec=3\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/app.serviceOutput:
[Unit]
Description=My App
[Service]
ExecStart=/usr/local/bin/app
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target
What this teaches: A minimal unit needs [Unit], [Service] with ExecStart, and [Install]; Restart=on-failure plus RestartSec makes the daemon self-healing. Type defaults to simple, so systemd tracks the foreground process you start; use Type=forking only for daemons that background themselves.
sudo systemctl daemon-reload && sudo systemctl enable --now app.serviceOutput:
Created symlink /etc/systemd/system/multi-user.target.wants/app.service → /etc/systemd/system/app.service.
$ systemctl is-active app.service
active
What this teaches: daemon-reload makes systemd re-read unit files; enable wires it into the boot target; --now starts it immediately too.
Output:
What this teaches: journalctl -u filters the journal to one unit's logs; stdout/stderr of the service land here automatically with no logfile config.
How It Works
systemd is PID 1 and the service manager: it parses unit files into an in-memory dependency graph, starts units in topological order, and supervises them afterwards. A service unit declares what to run (ExecStart), when (After=, WantedBy=), and how failures are handled (Restart=). This declarative model replaces fragile init scripts with state systemd can reason about.
Restart policies are the reliability core. on-failure restarts on crashes but not clean exits; always restarts regardless; RestartSec paces attempts to avoid hot loops. Combined with the journal capturing all output, a well-written unit is observable and self-healing without extra tooling.
Enablement is separate from starting: enable creates symlinks in .wants directories so the unit starts at boot, while start affects only now. systemctl is-enabled versus is-active answers the two different questions of 'will it start' and 'is it running'.
Progress So Far
Testing & Validation
sudo kill -9 $(systemctl show -p MainPID --value app.service); sleep 4; systemctl is-active app.serviceAfter killing the process you should see active again within seconds, proving auto-restart works.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Unit not found after creating the file | daemon-reload was skipped | Run systemctl daemon-reload after every unit file change |
| Service fails with exit 203 | ExecStart path wrong or not executable | Use absolute paths and chmod +x the binary |
| Restart loop spams the journal | Restart=always with an instantly-crashing ExecStart | Add RestartSec=5 and StartLimitIntervalSec/StartLimitBurst to pace attempts |
| Enabled but not starting at boot | WantedBy target wrong or enable created no symlink | Check systemctl is-enabled and that [Install] names multi-user.target |
| Environment variables missing | Units do not inherit your shell environment | Set them with Environment= lines or EnvironmentFile= |
| journalctl -u shows nothing | StandardOutput redirected or syslog forwarding only | Keep default journal capture; check with systemctl status for recent lines |
Troubleshooting
- Unit not found: run
systemctl daemon-reloadafter creating the file. - Service flaps: add
StartLimitIntervalSec/StartLimitBurstto avoid restart loops. - No logs: ensure the program writes to stdout/stderr, which journald captures.
Extension Ideas
- Pair the service with a systemd timer for scheduled runs.
- Sandbox it with directives like
ProtectSystem=strict. - Forward journald to a central log host (see Log Management).
Key Results
- Authored a custom unit that auto-restarts within ~3 seconds of failure.
- Enabled the service to start on boot reliably.
- Captured all service output in journald.
- Proved self-healing with a kill-and-recover test.
Key Takeaways
- Units declare what, when, and how-failures-handled; systemd supervises.
- daemon-reload after every unit edit; enable for boot, start for now.
- Restart=on-failure plus RestartSec gives self-healing services.
- journalctl -u is the log view; no logfile plumbing needed.
- Absolute ExecStart paths and Environment= lines prevent classic failures.
Practice Challenges
Challenge 1: Self-healing app
Goal: write a unit for a script that crashes randomly, prove with journalctl that systemd restarted it each time, and count restarts.
Hint: script exits 1 on a coin flip; watch systemctl show -p NRestarts.
Challenge 2: Dependency ordering
Goal: make app.service start only after network-online.target and a postgres.service, then verify the order in the boot graph.
Hint: After= plus Wants=, verified with systemd-analyze critical-chain.
Frequently Asked Questions
How do I fix 'Unit not found after creating the file'?
Likely cause: daemon-reload was skipped. Fix: Run systemctl daemon-reload after every unit file change.
How do I fix 'Service fails with exit 203'?
Likely cause: ExecStart path wrong or not executable. Fix: Use absolute paths and chmod +x the binary.
How do I fix 'Restart loop spams the journal'?
Likely cause: Restart=always with a instantly-crashing ExecStart. Fix: Add RestartSec=5 and StartLimitIntervalSec/StartLimitBurst to pace attempts.
How do I fix 'Enabled but not starting at boot'?
Likely cause: WantedBy target wrong or enable created no symlink. Fix: Check systemctl is-enabled and that [Install] names multi-user.target.
How do I fix 'Environment variables missing'?
Likely cause: Units do not inherit your shell environment. Fix: Set them with Environment= lines or EnvironmentFile=.
Cite This Page
Citation formats
Plain text
Taki Sadik, "Service Management with systemd: Units, Logs, and Auto-Restart", TYFSADIK, https://tyfsadik.org/blog/labs/linux-server/service-management (accessed 2026-10-22).
BibTeX
@misc{sadik-service-management,
author = {Sadik, MD. Taki Yasir Faraji},
title = {Service Management with systemd: Units, Logs, and Auto-Restart},
year = {2026},
url = {https://tyfsadik.org/blog/labs/linux-server/service-management},
urldate = {2026-10-22}
}
APA
Sadik, M. T. Y. F. (2026). Service Management with systemd: Units, Logs, and Auto-Restart. TYFSADIK. https://tyfsadik.org/blog/labs/linux-server/service-management