File Filters and Text Processing with Core Tools
I built one-line log analytics by chaining filter tools: cut to extract fields, sort and uniq to count occurrences, and head to rank. The classic sort | uniq -c | sort -rn idiom turned raw logs into top-N reports without a script.
Objective & Context
Filters are the workhorses of log triage. This lab assembles the standard idioms for field extraction and frequency analysis, the manual counterpart to the Python parsing and SIEM labs.
Learning Objectives
- Extract fields from delimited logs with cut.
- Sort and count unique values to build frequency tables with sort and uniq.
- Follow live logs with tail -f and filter the stream with grep.
- Count matches and lines with wc and grep -c.
- Chain filters into one-pass pipelines for common triage questions.
Environment & Prerequisites
Prerequisites
- Any Linux shell with GNU coreutils (preinstalled on all major distributions).
- A sample access or auth log; generate one or use /var/log/syslog.
- Know your log's field delimiter (spaces for Apache/Nginx combined logs).
- The redirection and pipelines lab pairs well with this one.
Step-by-Step Execution
1. Top source IPs in an access log
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | headExpected output:
842 203.0.113.45
311 198.51.100.7
96 192.0.2.14What this teaches: cut -d' ' -f1 pulls the client IP field; sort groups identical IPs; uniq -c counts each group; sort -rn orders by count so the top talkers surface first. This five-tool chain is the classic frequency analysis idiom.
2. Tail-follow with filtering
tail -f /var/log/syslog | grep -i errorExpected output (live stream):
Jun 17 09:14:02 srv01 kernel: [ 3.21] usb 1-2: error -71
Jun 17 09:14:09 srv01 nginx[812]: [error] upstream timed outWhat this teaches: tail -f keeps the file descriptor open and prints appended lines; piping into grep -i filters case-insensitively so only error lines reach your terminal while the log keeps flowing.
3. Count matching lines
grep -c 401 access.logExpected output:
27What this teaches: grep -c counts matching lines without printing them, which is faster and quieter than piping to wc -l when all you need is the number.
How It Works
Filters follow the Unix text contract: they read lines on stdin and write lines on stdout, which makes them freely composable. cut selects fields by delimiter, sort orders lines, uniq collapses adjacent duplicates (which is why sort must come first), and head/tail window the result. Each tool is deliberately narrow; the pipeline is where the analysis happens.
The frequency idiom (extract, sort, uniq -c, sort -rn, head) appears everywhere from access-log triage to CSV exploration because it answers 'what is most common' in a single pass. Understanding why uniq needs sorted input prevents the classic bug of undercounted groups.
tail -f works by holding the file open and reading new bytes as the writer appends them. Combined with grep it becomes a live alerting feed. When logrotate renames the file underneath you, plain -f keeps following the old descriptor, so switch to -F (follow by name with retry) to survive rotation.
Validation & Testing
Run the top-N pipeline against a known log and verify the counts match a manual spot-check. Pass criteria: correct field extraction, accurate frequency counts, and a ranked top-N list.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| uniq -c undercounts obvious duplicates | uniq only collapses adjacent lines; input was not sorted | Sort before uniq, or use sort -u when you only need distinct values |
| cut returns whole lines instead of fields | The delimiter flag does not match the actual separator | Inspect with cat -A to see tabs (^I) versus spaces, then set -d accordingly |
| tail -f stops after log rotation | The original file was renamed; tail still follows the old descriptor | Use tail -F (capital) which retries and follows the filename |
| sort -n misorders numbers like 10 before 9 | Sort defaulted to lexicographic order | Add -n for numeric sort, -r to reverse |
| Pipeline prints nothing but the log has data | An early stage filtered everything out (wrong field number or pattern) | Test each stage separately by truncating the pipeline |
| grep -c counts lines, not occurrences | A line with three matches still counts once | Use grep -o pattern | wc -l to count every occurrence |
Advanced: Troubleshooting
- Wrong field: confirm the delimiter; some logs use tabs, not spaces.
- uniq misses duplicates: uniq only collapses adjacent lines, so sort first.
- Huge files slow: pre-filter with grep before sorting.
Key Results
- Produced top-N source reports from raw logs in one line.
- Counted error and status-code occurrences instantly.
- Followed live logs filtered to relevant events.
- Avoided scripting for tasks the filter idioms already solve.
Key Takeaways
- cut, sort, uniq, head form the standard frequency-analysis chain.
- uniq requires sorted input to count correctly.
- tail -F survives log rotation; plain -f does not.
- grep -c counts matching lines; grep -o | wc -l counts total matches.
- Test pipelines stage by stage when output looks wrong.
Practice Challenges
Challenge 1: Top 404s by URL
Goal: from an access log, list the 10 most requested URLs that returned 404.
Hint: filter status 404 first with awk or grep, then extract field 7 and run the frequency chain.
Challenge 2: Live error watch
Goal: follow /var/log/syslog and print only lines containing error, warning, or fail, with timestamps preserved.
Hint: tail -F plus grep -i with an alternation pattern.