You will manage existing services and author a custom systemd unit that restarts automatically on failure. By the end you will have a resilient daemon with logs in journald and a clean enable-on-boot configuration.

Learning Objectives

  • Start, stop, enable, and inspect services with systemctl.
  • Write a custom unit with automatic restart.
  • Read service logs with journalctl.
  • Time: ~2 hours · Difficulty: Intermediate · Prereqs: a systemd-based Linux server.

Architecture Overview

Environment Setup

You will need: a Linux server with systemd and a simple program or script to run as a service.

Before you begin: confirm systemd is the init system with systemctl -version.

Prerequisites

  • A systemd-based Linux distribution (any modern Debian, Ubuntu, Fedora, RHEL).
  • Root or sudo access to install unit files.
  • A test binary or script to run as a service (/usr/local/bin/app used here).
  • Basic comfort with the terminal and text editors.

Step-by-Step Execution

The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run.

01
Inspect an existing service

Understanding state and dependencies is the basis of all service management.

systemctl status ssh --no-pager && systemctl list-dependencies ssh

Output (first 14 lines):

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
     Active: active (running) since Sat 2026-11-14 17:32:04 EST; 14ms ago
TriggeredBy: ● ssh.socket
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 457 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 458 (sshd)
      Tasks: 1 (limit: 7040)
     Memory: 1.1M (peak: 1.5M)
        CPU: 14ms
     CGroup: /docker/e18ffc53785729c64117a4ec9895a6a9cedb99e3dbbf9d7b31d42a665b4cd81f/system.slice/ssh.service
             └─458 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
ssh.service

What this teaches: status shows state, main PID, and recent logs; list-dependencies reveals ordering, which explains why a service waits at boot. Note: the unit is named ssh.service on Debian/Ubuntu but sshd.service on RHEL/Fedora; adjust the command to your distro.

02
Author a custom unit with auto-restart

The Restart directive makes the daemon self-heal after a crash, key for reliability.

printf '[Unit]\nDescription=My App\n[Service]\nExecStart=/usr/local/bin/app\nRestart=on-failure\nRestartSec=3\n[Install]\nWantedBy=multi-user.target\n' | sudo tee /etc/systemd/system/app.service
[ROOT REQUIRED] Creates a restart-on-failure service unit.

Output:

[Unit]
Description=My App
[Service]
ExecStart=/usr/local/bin/app
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target

What this teaches: A minimal unit needs [Unit], [Service] with ExecStart, and [Install]; Restart=on-failure plus RestartSec makes the daemon self-healing. Type defaults to simple, so systemd tracks the foreground process you start; use Type=forking only for daemons that background themselves.

03
Reload, enable, and start
sudo systemctl daemon-reload && sudo systemctl enable --now app.service

Output:

Created symlink /etc/systemd/system/multi-user.target.wants/app.service → /etc/systemd/system/app.service.
$ systemctl is-active app.service
active

What this teaches: daemon-reload makes systemd re-read unit files; enable wires it into the boot target; --now starts it immediately too.

04
Read the logs

Output:

# journalctl -u app.service -n 5 --no-pager
Nov 14 17:32:05 srv01 systemd[1]: Started app.service - My App. Nov 14 17:32:05 srv01 app[522]: app started, listening on :8080

What this teaches: journalctl -u filters the journal to one unit's logs; stdout/stderr of the service land here automatically with no logfile config.

How It Works

systemd is PID 1 and the service manager: it parses unit files into an in-memory dependency graph, starts units in topological order, and supervises them afterwards. A service unit declares what to run (ExecStart), when (After=, WantedBy=), and how failures are handled (Restart=). This declarative model replaces fragile init scripts with state systemd can reason about.

Restart policies are the reliability core. on-failure restarts on crashes but not clean exits; always restarts regardless; RestartSec paces attempts to avoid hot loops. Combined with the journal capturing all output, a well-written unit is observable and self-healing without extra tooling.

Enablement is separate from starting: enable creates symlinks in .wants directories so the unit starts at boot, while start affects only now. systemctl is-enabled versus is-active answers the two different questions of 'will it start' and 'is it running'.

Progress So Far

Testing & Validation

sudo kill -9 $(systemctl show -p MainPID --value app.service); sleep 4; systemctl is-active app.service

After killing the process you should see active again within seconds, proving auto-restart works.

Troubleshooting

SymptomLikely causeFix
Unit not found after creating the filedaemon-reload was skippedRun systemctl daemon-reload after every unit file change
Service fails with exit 203ExecStart path wrong or not executableUse absolute paths and chmod +x the binary
Restart loop spams the journalRestart=always with an instantly-crashing ExecStartAdd RestartSec=5 and StartLimitIntervalSec/StartLimitBurst to pace attempts
Enabled but not starting at bootWantedBy target wrong or enable created no symlinkCheck systemctl is-enabled and that [Install] names multi-user.target
Environment variables missingUnits do not inherit your shell environmentSet them with Environment= lines or EnvironmentFile=
journalctl -u shows nothingStandardOutput redirected or syslog forwarding onlyKeep default journal capture; check with systemctl status for recent lines
Troubleshooting
  • Unit not found: run systemctl daemon-reload after creating the file.
  • Service flaps: add StartLimitIntervalSec/StartLimitBurst to avoid restart loops.
  • No logs: ensure the program writes to stdout/stderr, which journald captures.

Extension Ideas

  • Pair the service with a systemd timer for scheduled runs.
  • Sandbox it with directives like ProtectSystem=strict.
  • Forward journald to a central log host (see Log Management).

Key Results

  • Authored a custom unit that auto-restarts within ~3 seconds of failure.
  • Enabled the service to start on boot reliably.
  • Captured all service output in journald.
  • Proved self-healing with a kill-and-recover test.

Key Takeaways

  • Units declare what, when, and how-failures-handled; systemd supervises.
  • daemon-reload after every unit edit; enable for boot, start for now.
  • Restart=on-failure plus RestartSec gives self-healing services.
  • journalctl -u is the log view; no logfile plumbing needed.
  • Absolute ExecStart paths and Environment= lines prevent classic failures.

Practice Challenges

Challenge 1: Self-healing app

Goal: write a unit for a script that crashes randomly, prove with journalctl that systemd restarted it each time, and count restarts.

Hint: script exits 1 on a coin flip; watch systemctl show -p NRestarts.

Challenge 2: Dependency ordering

Goal: make app.service start only after network-online.target and a postgres.service, then verify the order in the boot graph.

Hint: After= plus Wants=, verified with systemd-analyze critical-chain.

Frequently Asked Questions

How do I fix 'Unit not found after creating the file'?

Likely cause: daemon-reload was skipped. Fix: Run systemctl daemon-reload after every unit file change.

How do I fix 'Service fails with exit 203'?

Likely cause: ExecStart path wrong or not executable. Fix: Use absolute paths and chmod +x the binary.

How do I fix 'Restart loop spams the journal'?

Likely cause: Restart=always with a instantly-crashing ExecStart. Fix: Add RestartSec=5 and StartLimitIntervalSec/StartLimitBurst to pace attempts.

How do I fix 'Enabled but not starting at boot'?

Likely cause: WantedBy target wrong or enable created no symlink. Fix: Check systemctl is-enabled and that [Install] names multi-user.target.

How do I fix 'Environment variables missing'?

Likely cause: Units do not inherit your shell environment. Fix: Set them with Environment= lines or EnvironmentFile=.

Cite This Page

Citation formats

Plain text

Taki Sadik, "Service Management with systemd: Units, Logs, and Auto-Restart", TYFSADIK, https://tyfsadik.org/blog/labs/linux-server/service-management (accessed 2026-10-22).

BibTeX

@misc{sadik-service-management,
  author  = {Sadik, MD. Taki Yasir Faraji},
  title   = {Service Management with systemd: Units, Logs, and Auto-Restart},
  year    = {2026},
  url     = {https://tyfsadik.org/blog/labs/linux-server/service-management},
  urldate = {2026-10-22}
}

APA

Sadik, M. T. Y. F. (2026). Service Management with systemd: Units, Logs, and Auto-Restart. TYFSADIK. https://tyfsadik.org/blog/labs/linux-server/service-management