I built one-line log analytics by chaining filter tools: cut to extract fields, sort and uniq to count occurrences, and head to rank. The classic sort | uniq -c | sort -rn idiom turned raw logs into top-N reports without a script.

Objective & Context

Filters are the workhorses of log triage. This lab assembles the standard idioms for field extraction and frequency analysis, the manual counterpart to the Python parsing and SIEM labs.

Learning Objectives

  • Extract fields from delimited logs with cut.
  • Sort and count unique values to build frequency tables with sort and uniq.
  • Follow live logs with tail -f and filter the stream with grep.
  • Count matches and lines with wc and grep -c.
  • Chain filters into one-pass pipelines for common triage questions.

Environment & Prerequisites

Prerequisites

  • Any Linux shell with GNU coreutils (preinstalled on all major distributions).
  • A sample access or auth log; generate one or use /var/log/syslog.
  • Know your log's field delimiter (spaces for Apache/Nginx combined logs).
  • The redirection and pipelines lab pairs well with this one.

Step-by-Step Execution

The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run.

1. Top source IPs in an access log

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head

Output:

      9 203.0.113.45
      5 198.51.100.7
      3 192.0.2.14
      2 198.51.100.23

What this teaches: cut -d' ' -f1 pulls the client IP field; sort groups identical IPs; uniq -c counts each group; sort -rn orders by count so the top talkers surface first. This five-tool chain is the classic frequency analysis idiom.

2. Tail-follow with filtering

tail -f /var/log/syslog | grep -i error

Output (live stream):

2026-11-14T22:06:35.432800+00:00 srv01 nginx: [error] upstream timed out while reading response header
2026-11-14T22:06:35.456886+00:00 srv01 kernel: usb 1-2: device descriptor read/64, error -71

What this teaches: tail -f keeps the file descriptor open and prints appended lines; piping into grep -i filters case-insensitively so only error lines reach your terminal while the log keeps flowing.

3. Count matching lines

grep -c 401 access.log

Output:

3

What this teaches: grep -c counts matching lines without printing them, which is faster and quieter than piping to wc -l when all you need is the number.

How It Works

Filters follow the Unix text contract: they read lines on stdin and write lines on stdout, which makes them freely composable. cut selects fields by delimiter, sort orders lines, uniq collapses adjacent duplicates (which is why sort must come first), and head/tail window the result. Each tool is deliberately narrow; the pipeline is where the analysis happens.

The frequency idiom (extract, sort, uniq -c, sort -rn, head) appears everywhere from access-log triage to CSV exploration because it answers 'what is most common' in a single pass. Understanding why uniq needs sorted input prevents the classic bug of undercounted groups.

tail -f works by holding the file open and reading new bytes as the writer appends them. Combined with grep it becomes a live alerting feed. When logrotate renames the file underneath you, plain -f keeps following the old descriptor, so switch to -F (follow by name with retry) to survive rotation.

Validation & Testing

Run the top-N pipeline against a known log and verify the counts match a manual spot-check. Pass criteria: correct field extraction, accurate frequency counts, and a ranked top-N list.

Troubleshooting

SymptomLikely causeFix
uniq -c undercounts obvious duplicatesuniq only collapses adjacent lines; input was not sortedSort before uniq, or use sort -u when you only need distinct values
cut returns whole lines instead of fieldsThe delimiter flag does not match the actual separatorInspect with cat -A to see tabs (^I) versus spaces, then set -d accordingly
tail -f stops after log rotationThe original file was renamed; tail still follows the old descriptorUse tail -F (capital) which retries and follows the filename
sort -n misorders numbers like 10 before 9Sort defaulted to lexicographic orderAdd -n for numeric sort, -r to reverse
Pipeline prints nothing but the log has dataAn early stage filtered everything out (wrong field number or pattern)Test each stage separately by truncating the pipeline
grep -c counts lines, not occurrencesA line with three matches still counts onceUse grep -o pattern | wc -l to count every occurrence
Advanced: Troubleshooting
  • Wrong field: confirm the delimiter; some logs use tabs, not spaces.
  • uniq misses duplicates: uniq only collapses adjacent lines, so sort first.
  • Huge files slow: pre-filter with grep before sorting.

Key Results

  • Produced top-N source reports from raw logs in one line.
  • Counted error and status-code occurrences instantly.
  • Followed live logs filtered to relevant events.
  • Avoided scripting for tasks the filter idioms already solve.

Key Takeaways

  • cut, sort, uniq, head form the standard frequency-analysis chain.
  • uniq requires sorted input to count correctly.
  • tail -F survives log rotation; plain -f does not.
  • grep -c counts matching lines; grep -o | wc -l counts total matches.
  • Test pipelines stage by stage when output looks wrong.

Practice Challenges

Challenge 1: Top 404s by URL

Goal: from an access log, list the 10 most requested URLs that returned 404.

Hint: filter status 404 first with awk or grep, then extract field 7 and run the frequency chain.

Challenge 2: Live error watch

Goal: follow /var/log/syslog and print only lines containing error, warning, or fail, with timestamps preserved.

Hint: tail -F plus grep -i with an alternation pattern.

Frequently Asked Questions

How do I fix 'uniq -c undercounts obvious duplicates'?

Likely cause: uniq only collapses adjacent lines; input was not sorted. Fix: Sort before uniq, or use sort -u when you only need distinct values.

How do I fix 'cut returns whole lines instead of fields'?

Likely cause: The delimiter flag does not match the actual separator. Fix: Inspect with cat -A to see tabs (^I) versus spaces, then set -d accordingly.

How do I fix 'tail -f stops after log rotation'?

Likely cause: The original file was renamed; tail still follows the old descriptor. Fix: Use tail -F (capital) which retries and follows the filename.

How do I fix 'sort -n misorders numbers like 10 before 9'?

Likely cause: Sort defaulted to lexicographic order. Fix: Add -n for numeric sort, -r to reverse.

How do I fix 'Pipeline prints nothing but the log has data'?

Likely cause: An early stage filtered everything out (wrong field number or pattern). Fix: Test each stage separately by truncating the pipeline.

Cite This Page

Citation formats

Plain text

Taki Sadik, "File Filters and Text Processing with Core Tools", TYFSADIK, https://tyfsadik.org/blog/labs/linux-fundamentals/file-filters (accessed 2026-10-22).

BibTeX

@misc{sadik-file-filters,
  author  = {Sadik, MD. Taki Yasir Faraji},
  title   = {File Filters and Text Processing with Core Tools},
  year    = {2026},
  url     = {https://tyfsadik.org/blog/labs/linux-fundamentals/file-filters},
  urldate = {2026-10-22}
}

APA

Sadik, M. T. Y. F. (2026). File Filters and Text Processing with Core Tools. TYFSADIK. https://tyfsadik.org/blog/labs/linux-fundamentals/file-filters