Taki Sadik
Cybersecurity & IT Professional, North York, ON, Canada
Contact

Linux Process Management with ps, top, kill, and systemd

IntermediateLinux~3 hours
pstopsignalssystemd

I diagnosed and controlled processes using ps and top for inspection, signals for lifecycle control, nice/renice for scheduling priority, and systemd for service management. Sending the right signal in the right order stopped a runaway process cleanly instead of with a blunt SIGKILL.

Objective & Context

Process control is core incident response on a host. This lab covers reading process state, the signal model (TERM before KILL), priority tuning, and systemd as the modern init/service manager, feeding the IR and sysadmin labs.

Learning Objectives

  • Read process state, CPU, and memory columns in ps output.
  • Choose the right signal: TERM for graceful, KILL as a last resort.
  • Send signals by PID and by name with kill, pkill, and killall.
  • Adjust scheduling priority with nice and renice.
  • Manage services with systemctl: start, stop, restart, enable, status.

Environment & Prerequisites

Prerequisites

  • Linux with procps-ng and systemd (standard on modern distributions).
  • A test process to inspect and signal (stress-ng or a sleep loop).
  • A managed service unit to practice on (nginx, ssh, or cron).
  • Root or sudo for signaling other users' processes and managing services.

Step-by-Step Execution

1. Find the offending process

ps -eo pid,ppid,%cpu,%mem,comm --sort=-%cpu | head

What this teaches: -e selects every process; -o picks exact columns so output stays parseable; --sort=-%cpu orders by CPU descending, surfacing the offender at the top without eyeballing hundreds of rows.

2. Signal it gracefully, then forcefully

kill -TERM 4412 ; sleep 5 ; kill -KILL 4412 2>/dev/null

Expected output:

(no output if the process exits gracefully)
$ ps -p 4412
  PID TTY          TIME CMD
(no such process: terminated)

What this teaches: SIGTERM asks the process to clean up and exit; SIGKILL (-KILL) cannot be caught and the kernel reaps the process immediately. TERM first is both polite and safer for data integrity.

3. Manage a service with systemd

systemctl restart nginx && systemctl status nginx --no-pager

Expected output:

nginx.service: active (running)
Main PID: 812 (nginx)

What this teaches: systemctl is the control plane for the init system: restart bounces the service (it prints nothing on success), and status shows state, main PID, and recent log lines in one view.

4. Tune scheduling priority with nice and renice

nice -n 10 ./heavy-job.sh & renice -n 5 -p $!

Expected output:

4412 (process ID) old priority 10, new priority 5

What this teaches: nice -n 10 starts a process with lowered priority (higher niceness means a smaller CPU share); renice adjusts a running process by PID, and $! is the PID of the backgrounded job. Only root may set negative niceness to raise priority. Priority only matters when processes actually compete for CPU, which is why a lone busy loop shows no difference.

How It Works

A process is a running program plus kernel state: memory mappings, open file descriptors, and scheduling metadata, all visible under /proc/PID. ps formats that state into columns; STAT codes like R (running), S (sleeping), and Z (zombie) describe what the process is doing right now, while %CPU and %MEM quantify its footprint.

Signals are the kernel's inter-process messaging. SIGTERM (15) is a polite request the program can handle to flush buffers and exit; SIGKILL (9) bypasses the program entirely and the kernel destroys it. The TERM-before-KILL discipline exists because killing a database mid-write corrupts data, while asking first lets it checkpoint.

systemd sits above individual processes as the service manager: units describe how to start a daemon, its dependencies, and its restart policy. systemctl talks to PID 1 over D-Bus, which is why systemctl status can show the cgroup tree and journal logs together. Priority tuning with nice then decides how CPU time is shared when processes compete.

Validation & Testing

Start a CPU-bound test process, locate it, and stop it with the graceful-then-forceful sequence; restart a service and confirm it returns to active. Pass criteria: TERM is attempted before KILL and the service recovers cleanly under systemd.

Troubleshooting

SymptomLikely causeFix
kill reports no such process but ps showed itThe PID was reused or the process already exitedRe-run ps to confirm, and prefer pkill -f with a pattern over stale PIDs
Process ignores SIGTERM foreverIt masks the signal or is stuck in uninterruptible sleep (D state)Check STAT for D (wait for I/O); use SIGKILL only after TERM fails
kill: operation not permittedThe process belongs to another userUse sudo, or have the service owner signal it
systemctl restart hangsThe unit's ExecStop never completesInspect with systemctl status and journalctl -u, then consider TimeoutStopSec
Zombie processes accumulateThe parent is not reaping exited childrenFix or restart the parent; zombies are already dead and hold no resources
nice had no visible effectOnly one CPU-hungry process was running, so there was no contentionPriority matters when processes compete; test with two busy loops
Advanced: Troubleshooting
  • Process won't die: a process in uninterruptible sleep (D state) is waiting on I/O; address the I/O, not the signal.
  • Zombie processes: reap by fixing or restarting the parent, not by killing the zombie.
  • Service keeps dying: inspect journalctl -u service for the real failure.

Key Results

  • Identified the top CPU consumer instantly via sorted ps output.
  • Stopped runaway processes gracefully before forcing termination.
  • Managed service lifecycle reliably through systemd.
  • Diagnosed zombie and D-state cases correctly rather than blindly killing.

Key Takeaways

  • ps -eo with --sort surfaces resource hogs without manual scanning.
  • SIGTERM first for graceful shutdown; SIGKILL only as a last resort.
  • Uninterruptible (D-state) processes cannot be killed; wait out the I/O.
  • systemctl manages services; journalctl -u shows their logs.
  • nice/renice tune CPU sharing when processes actually compete.

Practice Challenges

Challenge 1: Graceful vs forceful

Goal: start a writer script that traps TERM to flush a file, then compare kill -TERM versus kill -KILL by checking whether the flush happened.

Hint: trap 'echo flushed >> out' TERM.

Challenge 2: Find the fork bomb's parent

Goal: given a runaway process tree, use ps --forest to identify the parent, then stop the whole tree cleanly.

Hint: pkill -P targets children of a PID.