Linux Redirection and Pipelines: stdin, stdout, stderr, and tee
I composed single-line data pipelines using redirection, pipes, and tee to filter, transform, and log output in one pass. Understanding the three standard streams let me capture stderr separately and split a pipeline to both a file and the screen.
Objective & Context
The Unix philosophy is small tools composed via streams. This lab masters file descriptors 0/1/2, pipe composition, and tee for branching, the glue behind every filter and scripting lab.
Learning Objectives
- Distinguish file descriptors 0 (stdin), 1 (stdout), and 2 (stderr).
- Redirect stdout and stderr to separate files or merge them.
- Compose multi-tool pipelines where each stage transforms the stream.
- Branch a stream to both a file and the next pipe stage with tee.
- Use process substitution and here-strings where pipes fall short.
Environment & Prerequisites
Prerequisites
- Any Bash shell with GNU coreutils (standard on all Linux distributions).
- A command that writes to both stdout and stderr (make, gcc, or curl -v).
- A log destination file for tee practice.
- No extra packages needed.
Step-by-Step Execution
1. Separate stdout and stderr
make 2> build-errors.log 1> build-out.logExpected output:
(no terminal output; both files created)
$ wc -l build-out.log build-errors.log
42 build-out.log
3 build-errors.logWhat this teaches: 2> captures only the error stream while 1> captures normal output; separating them keeps error triage clean. Order matters: 2>&1 must come after the stdout redirect to merge correctly.
2. Filter through a pipe
journalctl -u nginx | grep -i error | tail -20Expected output:
Jun 17 09:31:02 srv01 nginx[812]: [error] 1042#0: upstream timed out
(only the last 20 error lines shown)What this teaches: Each pipe stage filters the stream further: journalctl emits, grep keeps error lines, tail windows the last 20. Data flows left to right with no temporary files.
3. Split output with tee
dmesg | tee dmesg.log | grep -i failExpected output:
[ 3.21] usb 1-2: device descriptor read/64, error -71What this teaches: tee duplicates stdin to a file and stdout simultaneously, so you archive the raw dmesg output while grep shows only failures live. Add -a to append instead of overwriting.
4. Feed two live streams to one command with process substitution
diff <(sort hosts-a.txt) <(sort hosts-b.txt)Expected output:
2d1
< web02What this teaches: <(...) runs a command and presents its output as a readable file path (a /dev/fd entry), so tools like diff that demand two named file arguments can consume live command output. This covers the case pipes cannot: a command that needs two inputs at once. The >(...) form does the reverse, presenting a writable path.
How It Works
Every Linux process starts with three open streams: stdin (0) for input, stdout (1) for normal output, and stderr (2) for diagnostics. Redirection operators rebind these numbers to files before the command runs, which is why 2>file captures errors while leaving normal output on the terminal. The shell performs redirection itself; the command never knows its output went to a file.
A pipeline connects stdout of one process directly to stdin of the next through a kernel pipe buffer, so data streams through without touching disk. This is what makes long chains fast: each stage processes lines as they arrive. Merging with 2>&1 duplicates descriptor 2 onto wherever descriptor 1 currently points, which is why the ordering of redirections changes the result.
tee exists for the cases where you want both persistence and live processing: it reads stdin once and writes it to every named file plus stdout. Combined with process substitution, >(cmd), you can even fan one stream out to several processing commands at once, turning a linear pipeline into a small dataflow graph.
Validation & Testing
Run a command that emits both streams and confirm errors land in their own file while stdout is filtered. Pass criteria: stderr captured separately, the pipeline filters correctly, and tee writes the full stream to a file while passing it onward.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| 2>&1 produced an empty file | The merge was placed before the stdout redirect | Write cmd >file 2>&1 so stderr duplicates the already-redirected stdout |
| Pipeline shows output but the file is empty | Buffering: the writer holds data until its buffer fills or it exits | Use stdbuf -o0 or the tool's --line-buffered flag for live streams |
| tee overwrote yesterday's log | tee truncates by default like > | Add -a to append instead of overwriting |
| sudo cmd > /root/file gives permission denied | Redirection runs as your user, before sudo elevates | Use sudo tee: cmd | sudo tee /root/file >/dev/null |
| Pipe to head kills the producer with SIGPIPE | head exits early and the writer gets a broken pipe signal | Usually harmless; add 2>/dev/null if the noise bothers you |
| Here-string adds an unwanted trailing newline | <<< always appends a newline to the word | Use printf without \n piped in, or strip it downstream |
Advanced: Troubleshooting
- Errors not captured: redirect
2>explicitly; pipes only carry stdout by default. - Both streams merged: use
2>&1after the stdout redirect, order matters. - tee not writing: add
-ato append rather than overwrite.
Key Results
- Captured stdout and stderr to separate files in one command.
- Built filter pipelines that reduce noisy logs to relevant lines.
- Branched output to file and screen simultaneously with tee.
- Internalized the three standard streams for reliable scripting.
Key Takeaways
- Descriptors 0, 1, 2 are stdin, stdout, stderr; redirection rebinds them before the command runs.
- Separate streams for triage; merge with >file 2>&1 in that order when you want everything.
- Pipelines stream through kernel buffers with no temporary files.
- tee archives and displays simultaneously; -a appends.
- Redirection happens as your user, so sudo needs tee for privileged files.
Practice Challenges
Challenge 1: Silent build log
Goal: run a build so normal output goes to build-out.log, errors go to build-errors.log, and your terminal shows only a one-line summary.
Hint: redirect both streams, then echo based on the exit code.
Challenge 2: Fan-out monitoring
Goal: send one journalctl stream to three places at once: a full archive file, a live error-only view, and a count of critical lines.
Hint: combine tee with process substitution >( ).