Server Task Scheduling with cron and systemd Timers
You will schedule a recurring server task with both cron and a systemd timer, adding logging and missed-run catch-up. By the end you will have observable automation that survives reboots and tells you when it fails.
Learning Objectives
- Write a logged cron entry with a sane environment.
- Create an equivalent systemd timer with Persistent catch-up.
- Alert on failure rather than failing silently.
- Time: ~2 hours · Difficulty: Beginner · Prereqs: a systemd-based server.
Architecture Overview
Environment Setup
You will need: a systemd server and a script to schedule (for example the restic backup from the backup lab).
Before you begin: use absolute paths in scheduled jobs; cron's environment is minimal.
Prerequisites
- Linux with cron and systemd; root or sudo for system timers.
- A backup or maintenance script to schedule (the restic lab pairs well).
- A log directory writable by the job (e.g. /var/log).
- Mail or webhook alerting for failures (see CLI email lab).
Step-by-Step Execution
Create the script the schedule will run. It uses absolute paths, logs with UTC timestamps, and exits nonzero on failure (via set -euo pipefail) so problems surface instead of passing silently.
CREATE THE BACKUP SCRIPT
sudo tee /usr/local/bin/backup.sh > /dev/null <<'EOF'
#!/bin/bash
set -euo pipefail
# Scheduled jobs have no interactive shell: restic needs its password from the
# environment. Exporting it here covers both the cron entry and the systemd unit.
export RESTIC_PASSWORD_FILE=/root/.restic-pass
echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') starting backup"
/usr/bin/restic -r /backup/repo backup /etc /srv --tag daily
echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') backup finished"
EOF
sudo chmod +x /usr/local/bin/backup.shInstall the cron entry into root's crontab. The job writes to /var/log/backup.log and reads system paths, and a plain user crontab cannot write to /var/log, so sudo crontab is required here; for a non-root job, point the log at a user-writable path instead. The grep -v filter makes the install idempotent: re-running it never duplicates the line.
INSTALL THE CRON ENTRY (IDEMPOTENT)
( sudo crontab -l 2>/dev/null | grep -v -F '/usr/local/bin/backup.sh'; echo "30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1" ) | sudo crontab -
Expected output:
(no output on success; verify with: sudo crontab -l)
30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1What this teaches: Preserving the existing crontab while appending avoids wiping other jobs; filtering the old line first keeps re-runs idempotent; redirecting both streams to a log makes failures visible.
Persistent=true runs a missed job after downtime, which plain cron will not do. A timer never runs a script directly: it activates a same-named .service unit, so create backup.service first. flock -n in ExecStart makes an overrunning backup exit instead of stacking a second copy on top of it.
CREATE THE BACKUP.SERVICE UNIT
sudo tee /etc/systemd/system/backup.service > /dev/null <<'EOF'
[Unit]
Description=Daily backup job
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/flock -n /run/backup.lock /usr/local/bin/backup.sh
EOF
sudo systemd-analyze verify /etc/systemd/system/backup.service && echo "service OK"Environment=RESTIC_PASSWORD_FILE=/root/.restic-pass under [Service].Now create the timer that activates it:
printf '[Timer]\nOnCalendar=*-*-* 02:30:00\nPersistent=true\n[Install]\nWantedBy=timers.target\n' | sudo tee /etc/systemd/system/backup.timerExpected output:
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
[Install]
WantedBy=timers.targetWhat this teaches: OnCalendar expresses the schedule; Persistent=true replays missed runs after downtime, the key reliability advantage over cron.
Expected output:
What this teaches: daemon-reload picks up the new timer; enable --now activates it; list-timers confirms the next trigger time.
How It Works
On servers, scheduling is infrastructure: backups, rotations, cert renewals, and health checks all run unattended. cron remains the lingua franca because every Unix has it and the five-field syntax is universal, but its failure modes (silent failures, no catch-up, minimal environment) are exactly what cause 3 AM pages.
The operational discipline matters more than the tool: every scheduled job logs with timestamps, alerts on failure (never on success), is idempotent so overlapping runs are safe, and holds a lock (flock) so a slow run does not stack. These four habits turn scheduling from a hope into a system.
systemd timers add the missing reliability pieces: journal logging by default, dependency ordering (run after the database is up), calendar and monotonic triggers, and Persistent catch-up. For server workloads these advantages compound, which is why new automation should default to timers.
Progress So Far
Testing & Validation
sudo systemctl start backup.service && journalctl -u backup.service -n 5 --no-pagerYou should see the job run and log output. If the timer also appears in list-timers with a future run, scheduling is working.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Job runs but produces nothing | Output lost: no redirection and no mail configured | Redirect to a log file and check it; add MAILTO as a backup |
| Timer fired during a deploy window | Schedule collides with maintenance | Use RandomizedDelaySec to spread load and document blackout windows |
| Overlapping runs corrupt state | Job outlasts its interval | Wrap ExecStart with flock -n or implement locking in the script |
| Cron PATH breaks the script | Minimal environment strikes again | Absolute paths everywhere; set PATH explicitly at the top of the crontab |
| Missed runs after weekend downtime | cron never catches up | Use a systemd timer with Persistent=true for anything that must not be skipped |
| Timezone confusion in schedules | Cron uses system local time | Keep servers on UTC and express schedules in UTC to avoid DST surprises |
Troubleshooting
- Cron job runs in shell but not scheduled: cron has a minimal PATH; use absolute paths and set variables.
- Timer never fires: confirm it is enabled and the
OnCalendarsyntax withsystemd-analyze calendar. - Missed runs: set
Persistent=trueso the job catches up after downtime.
Extension Ideas
- Add an
OnFailure=unit that sends a notification. - Schedule the restic backups from the backup lab.
- Randomize start with
RandomizedDelaySecto avoid thundering herds.
Key Results
- Scheduled a task via both cron and a systemd timer.
- Captured all job output to logs for observability.
- Enabled missed-run catch-up with Persistent timers.
- Validated the schedule with list-timers and a manual run.
Key Takeaways
- Log every scheduled job; alert on failure, not success.
- Make jobs idempotent and locked against overlap.
- cron is universal; systemd timers add logging, deps, and catch-up.
- Persistent=true replays missed runs after downtime.
- UTC schedules avoid daylight-saving surprises.
- Set MAILTO so job output reaches your inbox; silent failures stop being silent.
- Use full paths in jobs because the cron PATH is minimal.
Practice Challenges
Challenge 1: Observable backup
Goal: schedule the restic backup from the backup lab via a systemd timer with logging, failure alerting, and flock protection.
Hint: the timer calls a wrapper script that handles all four.
Challenge 2: Cron audit
Goal: inventory every cron entry and timer on a host, document what each does and who owns it, and flag any without logging.
Hint: /etc/cron*, /var/spool/cron, systemctl list-timers.