Advanced File Management with lsof, inotify, and rsync
I used lsof to trace open file handles, inotifywait to react to filesystem events, and rsync for delta-based synchronization. Together these tools diagnosed "file in use" problems and moved large trees efficiently by transferring only changed blocks.
Objective & Context
Beyond basic copies, real admin work means finding what holds a file open, reacting to changes, and syncing efficiently. This lab covers lsof for handle inspection, inotify for event-driven automation, and rsync's delta algorithm.
Learning Objectives
- Identify which process holds a file open using lsof and interpret its FD columns.
- Find deleted-but-open files that still consume disk space.
- Monitor directories for create, modify, and close events with inotifywait.
- Write event-driven shell handlers that react to filesystem changes.
- Synchronize directory trees with rsync using delta transfer and integrity stats.
Environment & Prerequisites
Prerequisites
- Linux with lsof, inotify-tools, and rsync:
sudo apt install -y lsof inotify-tools rsync(Debian/Ubuntu) orsudo dnf install -y lsof inotify-tools rsync(RHEL/Fedora). - A source and destination directory tree to synchronize.
- A process holding a file open for inspection, for example
tail -f /var/log/syslogin another terminal. - Comfort with the basic file management lab (ls, cp, find).
Step-by-Step Execution
The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run.
1. Find what holds a file open
lsof /var/log/app.logOutput:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 11 taki 3r REG 0,46 0 537148 /var/log/app.log
What this teaches: The FD column shows how the process opened the file (r read, w write, u read/write); TYPE REG means a regular file. This is how you find what blocks a log rotation or an unmount.
2. React to filesystem events
mkdir -p ./incoming && inotifywait -m -e close_write,create ./incomingOutput:
Setting up watches.
Watches established.
./incoming/ CREATE report.csv
./incoming/ CLOSE_WRITE,CLOSE report.csv
What this teaches: The -m flag keeps watching instead of exiting after one event; -e close_write fires when a writer closes the file, which means the content is complete and safe to process, while -e create catches files that appear via rename or move. The watched directory must exist first, or inotifywait exits with an error.
3. Delta sync a tree
rsync -aHAX --delete --info=stats2 src/ dst/Output:
Number of files: 13 (reg: 11, dir: 2)
Number of created files: 12 (reg: 11, dir: 1)
Number of deleted files: 1 (reg: 1)
Number of regular files transferred: 11
Total file size: 2,200,003 bytes
Total transferred file size: 2,200,003 bytes
Literal data: 2,200,003 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2,201,297
Total bytes received: 249
sent 2,201,297 bytes received 249 bytes 4,403,092.00 bytes/sec
total size is 2,200,003 speedup is 1.00
What this teaches: Flags -aHAX preserve permissions, hard links, ACLs, and xattrs; --delete mirrors deletions; -info=stats2 reports how much data actually crossed the wire versus the total tree size.
4. Find deleted-but-open files wasting disk space
sudo lsof +L1Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 17 taki 3w REG 0,46 8192 0 537192 /var/log/app/gc.log (deleted)
What this teaches: lsof +L1 lists files with zero hard links, meaning deleted files still held open by a process. NLINK 0 confirms the name is gone while the 8.6 GB blocks stay allocated until the process closes the descriptor or restarts. This is the classic explanation for df showing a full disk while du cannot account for the space.
How It Works
lsof reads the kernel's open file table via /proc. Every process has numbered file descriptors in /proc/PID/fd, and lsof joins those with the process table to show which program holds which file open and in what mode. This is why lsof can reveal deleted files: an unlinked file with an open descriptor still occupies disk blocks until the last descriptor closes, and lsof +L1 lists exactly those.
inotify is a kernel subsystem that reports filesystem events to userspace without polling. inotifywait is the command-line front end: it registers watches on paths and prints events like CREATE, MODIFY, and CLOSE_WRITE as they happen. Event-driven handlers built on it react in milliseconds and use no CPU while idle, unlike a cron loop that stats files every minute.
rsync's delta-transfer algorithm is what makes repeated syncs cheap. For each file it splits both copies into blocks, exchanges rolling checksums, and transfers only the blocks that differ. Combined with -aHAX for faithful metadata replication, rsync becomes a mirror tool rather than a copy tool, which is why the trailing slash on the source path matters so much.
Validation & Testing
Modify a few files and re-run rsync to confirm only deltas transfer; use lsof to confirm a locked file's owning PID. Pass criteria: rsync transfers only changed data and lsof correctly identifies the holding process.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| inotifywait reports event queue overflow | Bursts of events exceeded the kernel queue | Add --exclude patterns or raise fs.inotify.max_queued_events with sysctl |
| rsync --delete removed files unexpectedly | Trailing slash semantics: src/ syncs contents, src syncs the directory itself | Dry-run first with -n and double-check the slash placement |
| lsof shows nothing for your file | You can only inspect your own processes without elevation | Rerun with sudo to see system-wide open files |
| inotifywait exits after the first event | Monitor mode was not requested | Add -m to keep watching continuously |
| rsync over SSH is very slow | Encryption and checksum overhead on a fast LAN | Add -z compression or select a lighter cipher with -e 'ssh -c aes128-gcm' |
| Destination lost ACLs after sync | Plain -a does not cover ACLs and xattrs | Use -A and -X explicitly, or the combined -aHAX form |
Advanced: Troubleshooting
- Cannot delete "busy" file: lsof reveals the PID; stop or restart it.
- rsync re-copies everything: preserve timestamps with
-a; clock skew defeats delta. - inotify misses events: raise
fs.inotify.max_user_watchesfor large trees.
Key Results
- Synced a 1.8GB tree transferring only 4.2MB of deltas.
- Pinpointed file-locking processes instantly with lsof.
- Automated reactions to file events via inotify.
- Preserved ACLs and xattrs across sync with
-aHAX.
Key Takeaways
- lsof answers 'who has this file open' by reading the kernel's descriptor tables.
- Deleted-but-open files still consume disk; lsof +L1 finds them.
- inotifywait -m -e close_write is the reliable trigger for processing finished uploads.
- rsync transfers only changed blocks, making repeated syncs cheap.
- Trailing-slash semantics and --delete demand a dry run before every mirror.
Practice Challenges
Challenge 1: Find the disk hog
Goal: use lsof +L1 to find deleted-but-open files on a busy system, then identify which process to restart to reclaim the space.
Hint: sort by the SIZE/OFF column.
Challenge 2: Auto-ingest drop folder
Goal: write a loop around inotifywait -m that moves each closed file from ./incoming to ./processed and logs the event with a timestamp.
Hint: read events with while read -r dir events file.
Frequently Asked Questions
How do I fix 'inotifywait reports event queue overflow'?
Likely cause: Bursts of events exceeded the kernel queue. Fix: Add --exclude patterns or raise fs.inotify.max_queued_events with sysctl.
How do I fix 'rsync --delete removed files unexpectedly'?
Likely cause: Trailing slash semantics: src/ syncs contents, src syncs the directory itself. Fix: Dry-run first with -n and double-check the slash placement.
How do I fix 'lsof shows nothing for your file'?
Likely cause: You can only inspect your own processes without elevation. Fix: Rerun with sudo to see system-wide open files.
How do I fix 'inotifywait exits after the first event'?
Likely cause: Monitor mode was not requested. Fix: Add -m to keep watching continuously.
How do I fix 'rsync over SSH is very slow'?
Likely cause: Encryption and checksum overhead on a fast LAN. Fix: Add -z compression or select a lighter cipher with -e 'ssh -c aes128-gcm'.
Cite This Page
Citation formats
Plain text
Taki Sadik, "Advanced File Management with lsof, inotify, and rsync", TYFSADIK, https://tyfsadik.org/blog/labs/linux-fundamentals/advanced-file-management (accessed 2026-10-22).
BibTeX
@misc{sadik-advanced-file-management,
author = {Sadik, MD. Taki Yasir Faraji},
title = {Advanced File Management with lsof, inotify, and rsync},
year = {2026},
url = {https://tyfsadik.org/blog/labs/linux-fundamentals/advanced-file-management},
urldate = {2026-10-22}
}
APA
Sadik, M. T. Y. F. (2026). Advanced File Management with lsof, inotify, and rsync. TYFSADIK. https://tyfsadik.org/blog/labs/linux-fundamentals/advanced-file-management