Linux File Transfer with scp, rsync, and Command-Line Email
I moved files between hosts with scp for quick copies and rsync for resumable, verified, delta transfers, then wired CLI email for job notifications. Resumable rsync transfers survived interruptions that would have forced scp to restart from zero.
Objective & Context
Automation pipelines must move artifacts reliably and report results. This lab compares scp, sftp, and rsync transports and adds curl/mail-based notifications, the transfer layer for backups and deployments.
Learning Objectives
- Copy files quickly between hosts with scp.
- Resume interrupted transfers and verify integrity with rsync.
- Choose between scp, sftp, and rsync for a given transfer need.
- Send completion notifications via webhook or email from scripts.
- Secure transfers with key-based SSH authentication.
Environment & Prerequisites
Prerequisites
- Two SSH-reachable Linux hosts with rsync installed on both.
- SSH key authentication configured (see the SSH remote access lab).
- curl and a mail relay or webhook URL for the notification step.
- A large test file to demonstrate resume (1 GB works well).
Step-by-Step Execution
1. Quick copy with scp
scp report.tar.gz taki@nas:/backups/Expected output:
report.tar.gz 100% 512MB 84.2MB/s 00:06What this teaches: scp is the simplest secure copy: one command over SSH with a progress meter. It re-sends everything on interruption, which is why large files want rsync instead.
2. Resumable, verified transfer with rsync
rsync -avP --checksum big.img taki@nas:/backups/Expected output:
sending incremental file list
big.img
1,073,741,824 100% 118.20MB/s 0:00:08 (xfr#1, to-chk=0/1)What this teaches: -a preserves attributes, -v is verbose, -P combines --partial (keep interrupted files) with --progress, and --checksum verifies content by hash rather than size and time. Rerunning after an interruption resumes from the partial file instead of starting over. Resume comes from --partial inside -P, not from --checksum, which only adds verification.
3. Notify on completion
curl -s -X POST $WEBHOOK -d 'text=backup transfer complete'Expected output:
(webhook returns 200 OK; mail relay accepts the message)
{"ok": true}What this teaches: Automation must report outcomes: a curl POST to a webhook or a mailx message turns a silent transfer into an observable pipeline step that cron jobs and CI can rely on. $WEBHOOK must be exported in the shell first; in cron, define it inside the script since cron's environment is minimal.
How It Works
All three transports ride on SSH, which provides encryption and authentication, but they differ in smarts. scp copies byte streams with no resume and no delta logic. sftp adds an interactive file protocol (list, get, put, resume-aware clients) over the same SSH channel. rsync adds its delta-transfer algorithm plus attribute preservation, making it the right default for anything repeated or large.
Resume works because rsync keeps partial files and compares block checksums on rerun: already-correct blocks are skipped, so a 62%-complete transfer continues instead of restarting. --checksum goes further, verifying the final content matches the source even when sizes and timestamps already agree.
Notifications close the automation loop. A transfer inside a cron job or backup script that fails silently is worse than no automation, because it creates false confidence. A webhook POST or an email with the exit status attached makes success and failure equally visible.
Validation & Testing
Interrupt an rsync transfer and restart it to confirm it resumes rather than restarts, then verify the checksum matches. Pass criteria: rsync resumes from the breakpoint, checksums match, and the notification fires on success.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| scp stalls with no progress | A middlebox or the server throttles, or the disk filled mid-write | Check destination space with df -h and retry with rsync for resume |
| rsync: command not found on remote | rsync missing on one side of the transfer | Install rsync on both hosts; the protocol needs it at each end |
| Permission denied on destination | SSH user lacks write rights on the target path | Fix ownership on the destination or transfer to a writable staging dir |
| --checksum is very slow on huge files | Every block is hashed on both sides | Drop --checksum for routine syncs; keep it for integrity-critical copies |
| Webhook notification never arrives | Wrong URL, missing auth header, or blocked egress | Test with curl -v and check the exit code before trusting the pipeline |
| Transfer corrupts filenames with spaces | Shell quoting lost around the remote path | Quote the full remote spec: scp file 'user@host:/path with spaces/' |
Advanced: Troubleshooting
- Transfer restarts fully: use
-P(partial + progress) so rsync resumes. - Permission denied: confirm SSH key auth and destination write perms.
- Silent corruption: add
--checksumfor content verification, not just size/time.
Key Results
- Resumed interrupted transfers instead of restarting from zero.
- Verified transfer integrity with content checksums.
- Automated completion notifications via webhook/mail.
- Selected the right transport (scp vs rsync) per task.
Key Takeaways
- scp for quick one-shots; rsync for anything large, repeated, or resumable.
- rsync -avP --checksum resumes partial transfers and verifies integrity.
- All three transports inherit SSH encryption and key auth.
- Automated transfers must notify: silent success breeds false confidence.
- Quote remote paths that contain spaces.
Practice Challenges
Challenge 1: Resume drill
Goal: start a 2 GB rsync transfer, kill it at 50%, then rerun and confirm it resumes instead of restarting.
Hint: watch the progress meter's starting percentage.
Challenge 2: Verified deploy
Goal: script a transfer that rsyncs a build artifact, verifies its sha256 against the source, and posts success or failure to a webhook.
Hint: check $? after each stage.