Python System Administration Scripts with psutil and subprocess
I automated routine Linux administration with Python, using psutil for resource metrics and subprocess for safe command execution. Scheduled health and cleanup scripts replaced manual checks and surfaced disk and memory pressure before it caused outages.
Objective & Context
Repetitive admin tasks are error-prone when done by hand. This lab builds idempotent Python scripts that gather metrics, act on thresholds, and run under systemd timers, the toolkit that underpins the IR automation and vulnerability labs.
Learning Objectives
- Read CPU, memory, and disk metrics with psutil.
- Run shell commands safely with subprocess (no shell=True).
- Write alerting logic with thresholds and notifications.
- Schedule scripts with systemd timer and service units.
- Handle permissions correctly for privileged checks.
Environment & Prerequisites
Prerequisites
- A Linux host with
Python 3.11andpsutil(pip install psutil) systemdfor scheduling (timer + service unit); root for enabling the timer- Permissions appropriate to the resources being managed
- Assumed skill: basic Linux commands and Python functions
Step-by-Step Execution
1. Gather metrics with psutil
import psutil
def alert(disk, mem):
print(f"ALERT: disk {disk}% mem {mem}%")
disk = psutil.disk_usage("/").percent
mem = psutil.virtual_memory().percent
if disk > 85 or mem > 90:
alert(disk, mem)Expected output:
(no output; alert fires only over threshold)What this teaches: psutil reads from /proc, so the check is cheap and dependency-free; thresholds turn raw metrics into actionable signals. The alert function is a placeholder, wire it to your real notification path (mail, webhook, pager). Save this as /opt/health/health.py for the timer step below.
2. Run a command safely with subprocess
python -c "import subprocess; print(subprocess.run(['df','-h','/'],capture_output=True,text=True).stdout)"Expected output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 ...What this teaches: Passing the command as a list avoids the shell entirely, which eliminates shell-injection risk from interpolated arguments.
3. Schedule via systemd timer [ROOT REQUIRED]
sudo tee /etc/systemd/system/health.service > /dev/null <<'EOF'
[Unit]
Description=System health check
[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /opt/health/health.py
WorkingDirectory=/opt/health
User=root
EOF
sudo tee /etc/systemd/system/health.timer > /dev/null <<'EOF'
[Unit]
Description=Run health check hourly
[Timer]
OnCalendar=hourly
Persistent=true
[Install]
WantedBy=timers.target
EOFExpected output:
(no output; unit files written; needs sudo or a root shell)What this teaches: The service unit pins the interpreter path, working directory, and user so the script runs the same from the timer as it does by hand. Writing under /etc/systemd/system requires root; use a ~/.config/systemd/user/ timer instead if you have no sudo on the box.
sudo systemctl daemon-reload && sudo systemctl enable --now health.timer && systemctl list-timers health.timerExpected output:
NEXT UNIT ACTIVATES
2026-06-17 10:00:00 UTC health.timer health.serviceWhat this teaches: daemon-reload makes systemd pick up the new units, then the timer appearing in list-timers proves systemd accepted the schedule; enable --now both enables and starts it in one command.
How It Works
How It Works
psutil is a cross-platform /proc reader: it parses the same kernel interfaces that top and df use, but returns them as Python objects. That makes health checks a few lines of Python instead of fragile text parsing of command output.
subprocess.run with a list argument bypasses the shell, so arguments are passed literally to the program. shell=True reintroduces parsing, quoting, and injection hazards; the list form is the safe default for automation.
systemd timers replace cron for service-style scheduling: they support calendar and monotonic triggers, jitter, persistent catch-up after downtime, and their runs appear in the journal, which makes scheduled Python observable like any other service.
disk, mem, cpu"] --> TH{"over threshold?"} TH -->|"yes"| AL["alert()"] TH -->|"no"| OK["log ok"] SUB["subprocess.run([...])"] --> PS TMR["systemd timer"] --> SUB
Validation & Testing
Fill a test filesystem past the threshold and confirm the script alerts and runs cleanup, then verify the timer fires on schedule. Pass criteria: threshold logic triggers correctly, subprocess uses a list (no shell injection), and the timer runs reliably.
Advanced: Troubleshooting
- Shell injection risk: pass args as a list, never
shell=Truewith user input. - Timer not firing: check
systemctl status health.timerand journal logs. - Permission denied: run the service as a user with the needed access, not blanket root.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| psutil raises AccessDenied | reading another user's process info without permission | run as the owning user or root; scope checks to system-wide metrics instead |
| subprocess FileNotFoundError | the program is not on PATH in the timer's environment | use absolute paths in scheduled scripts; systemd units get a minimal PATH |
| timer never fires | the unit was enabled but not started, or the calendar expression is wrong | use enable --now and verify with systemctl list-timers; check systemd-analyze calendar |
| script works manually but fails from the timer | environment differences: cwd, env vars, or user | set WorkingDirectory= and User= in the service unit; log stderr to the journal |
| alerts fire constantly (flapping) | threshold crossed repeatedly around the boundary | add hysteresis or require N consecutive breaches before alerting |
Key Results
- Replaced manual checks with scheduled health monitoring.
- Surfaced disk/memory pressure before it caused service impact.
- Eliminated shell-injection risk via list-form subprocess calls.
- Ran idempotent cleanup automatically under systemd timers.
Key Takeaways
- psutil turns /proc into Python objects: no text parsing needed.
- Always pass subprocess commands as lists; avoid shell=True.
- Thresholds plus alerting turn metrics into action.
- systemd timers make scheduled scripts observable via the journal.
- Scheduled environments are minimal: use absolute paths and explicit units.
Practice Challenges
Challenge 1: Hysteresis alerter
Goal: Extend the disk check to alert only after three consecutive breaches and clear after three consecutive recoveries.
Hint: Keep a small state file or counter between runs; the timer invokes the script each time.
Challenge 2: Journal-friendly script
Goal: Make the script log structured key=value lines so journalctl queries can filter by metric.
Hint: Print to stdout from the service unit; systemd captures it into the journal automatically.