Taki Sadik
Cybersecurity & IT Professional, North York, ON, Canada
Contact

Advanced File Management with lsof, inotify, and rsync

IntermediateLinux~3 hours
lsofinotifywaitrsyncdelta

I used lsof to trace open file handles, inotifywait to react to filesystem events, and rsync for delta-based synchronization. Together these tools diagnosed "file in use" problems and moved large trees efficiently by transferring only changed blocks.

Objective & Context

Beyond basic copies, real admin work means finding what holds a file open, reacting to changes, and syncing efficiently. This lab covers lsof for handle inspection, inotify for event-driven automation, and rsync's delta algorithm.

Learning Objectives

  • Identify which process holds a file open using lsof and interpret its FD columns.
  • Find deleted-but-open files that still consume disk space.
  • Monitor directories for create, modify, and close events with inotifywait.
  • Write event-driven shell handlers that react to filesystem changes.
  • Synchronize directory trees with rsync using delta transfer and integrity stats.

Environment & Prerequisites

Prerequisites

  • Linux with lsof, inotify-tools, and rsync: sudo apt install -y lsof inotify-tools rsync (Debian/Ubuntu) or sudo dnf install -y lsof inotify-tools rsync (RHEL/Fedora).
  • A source and destination directory tree to synchronize.
  • A process holding a file open for inspection, for example tail -f /var/log/syslog in another terminal.
  • Comfort with the basic file management lab (ls, cp, find).

Step-by-Step Execution

1. Find what holds a file open

lsof /var/log/app.log

Expected output:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
tail     1842 taki    3r   REG  259,1    48210  131 /var/log/app.log

What this teaches: The FD column shows how the process opened the file (r read, w write, u read/write); TYPE REG means a regular file. This is how you find what blocks a log rotation or an unmount.

2. React to filesystem events

mkdir -p ./incoming && inotifywait -m -e close_write,create ./incoming

Expected output:

./incoming/ CREATE report.csv
./incoming/ CLOSE_WRITE,CLOSE report.csv

What this teaches: The -m flag keeps watching instead of exiting after one event; -e close_write fires when a writer closes the file, which means the content is complete and safe to process, while -e create catches files that appear via rename or move. The watched directory must exist first, or inotifywait exits with an error.

3. Delta sync a tree

rsync -aHAX --delete --info=stats2 src/ dst/

Expected output:

Number of regular files transferred: 12
Total transferred file size: 4.20M (of 1.8G total)

What this teaches: Flags -aHAX preserve permissions, hard links, ACLs, and xattrs; --delete mirrors deletions; -info=stats2 reports how much data actually crossed the wire versus the total tree size.

4. Find deleted-but-open files wasting disk space

sudo lsof +L1

Expected output:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
java     2210 svc     7w   REG  259,1 8589934592     0 /var/log/app/gc.log (deleted)

What this teaches: lsof +L1 lists files with zero hard links, meaning deleted files still held open by a process. NLINK 0 confirms the name is gone while the 8.6 GB blocks stay allocated until the process closes the descriptor or restarts. This is the classic explanation for df showing a full disk while du cannot account for the space.

How It Works

lsof reads the kernel's open file table via /proc. Every process has numbered file descriptors in /proc/PID/fd, and lsof joins those with the process table to show which program holds which file open and in what mode. This is why lsof can reveal deleted files: an unlinked file with an open descriptor still occupies disk blocks until the last descriptor closes, and lsof +L1 lists exactly those.

inotify is a kernel subsystem that reports filesystem events to userspace without polling. inotifywait is the command-line front end: it registers watches on paths and prints events like CREATE, MODIFY, and CLOSE_WRITE as they happen. Event-driven handlers built on it react in milliseconds and use no CPU while idle, unlike a cron loop that stats files every minute.

rsync's delta-transfer algorithm is what makes repeated syncs cheap. For each file it splits both copies into blocks, exchanges rolling checksums, and transfers only the blocks that differ. Combined with -aHAX for faithful metadata replication, rsync becomes a mirror tool rather than a copy tool, which is why the trailing slash on the source path matters so much.

Validation & Testing

Modify a few files and re-run rsync to confirm only deltas transfer; use lsof to confirm a locked file's owning PID. Pass criteria: rsync transfers only changed data and lsof correctly identifies the holding process.

Troubleshooting

SymptomLikely causeFix
inotifywait reports event queue overflowBursts of events exceeded the kernel queueAdd --exclude patterns or raise fs.inotify.max_queued_events with sysctl
rsync --delete removed files unexpectedlyTrailing slash semantics: src/ syncs contents, src syncs the directory itselfDry-run first with -n and double-check the slash placement
lsof shows nothing for your fileYou can only inspect your own processes without elevationRerun with sudo to see system-wide open files
inotifywait exits after the first eventMonitor mode was not requestedAdd -m to keep watching continuously
rsync over SSH is very slowEncryption and checksum overhead on a fast LANAdd -z compression or select a lighter cipher with -e 'ssh -c aes128-gcm'
Destination lost ACLs after syncPlain -a does not cover ACLs and xattrsUse -A and -X explicitly, or the combined -aHAX form
Advanced: Troubleshooting
  • Cannot delete "busy" file: lsof reveals the PID; stop or restart it.
  • rsync re-copies everything: preserve timestamps with -a; clock skew defeats delta.
  • inotify misses events: raise fs.inotify.max_user_watches for large trees.

Key Results

  • Synced a 1.8GB tree transferring only 4.2MB of deltas.
  • Pinpointed file-locking processes instantly with lsof.
  • Automated reactions to file events via inotify.
  • Preserved ACLs and xattrs across sync with -aHAX.

Key Takeaways

  • lsof answers 'who has this file open' by reading the kernel's descriptor tables.
  • Deleted-but-open files still consume disk; lsof +L1 finds them.
  • inotifywait -m -e close_write is the reliable trigger for processing finished uploads.
  • rsync transfers only changed blocks, making repeated syncs cheap.
  • Trailing-slash semantics and --delete demand a dry run before every mirror.

Practice Challenges

Challenge 1: Find the disk hog

Goal: use lsof +L1 to find deleted-but-open files on a busy system, then identify which process to restart to reclaim the space.

Hint: sort by the SIZE/OFF column.

Challenge 2: Auto-ingest drop folder

Goal: write a loop around inotifywait -m that moves each closed file from ./incoming to ./processed and logs the event with a timestamp.

Hint: read events with while read -r dir events file.