Linux Log Management with journald, rsyslog, and logrotate
You will query journald, forward logs to a central rsyslog server, and rotate files so disks never fill. By the end you will have searchable, retained, centralized logs, the foundation for monitoring and the SOC labs.
Learning Objectives
- Filter and follow logs with journalctl.
- Forward logs to a central server over rsyslog.
- Rotate and retain logs with logrotate.
- Time: ~3 hours · Difficulty: Intermediate · Prereqs: a server (and optionally a second host as collector).
Architecture Overview
Environment Setup
You will need: systemd-journald (default), rsyslog, and logrotate.
Before you begin: if forwarding, have a reachable collector listening on TCP/514.
Prerequisites
- A systemd-based Linux host with journald and rsyslog installed.
- Root or sudo access for rsyslog config and logrotate.
- A central log server address for the forwarding step (or a second VM).
- A service generating logs (ssh used in the examples).
Step-by-Step Execution
Targeted filters turn a flood of entries into the few that matter for triage.
journalctl -p info -S "2026-06-17 00:00:00" -u ssh --no-pager-p err would hide exactly the lines you are hunting.
Expected output:
Jun 17 09:02:11 srv01 sshd[2101]: Failed password for invalid user admin from 203.0.113.45
Jun 17 09:02:14 srv01 sshd[2101]: Failed password for invalid user admin from 203.0.113.45What this teaches: -p filters by priority (err and higher would miss these auth lines, which are info), -S sets the start time, -u scopes to one unit: three filters turn a flood into the exact lines triage needs. Match the priority to what the service actually logs.
Centralized logs survive a compromised host and enable cross-host correlation.
echo '*.* @@192.168.1.60:514' | sudo tee /etc/rsyslog.d/90-forward.conf && sudo systemctl restart rsyslog
Expected output:
*.* @@192.168.1.60:514
(rsyslog restarted; verify with: systemctl is-active rsyslog)What this teaches: A one-line rsyslog rule forwards everything over TCP (@@) to the collector; centralized logs survive a compromised or dead host.
The collector (192.168.1.60) must actually be listening, or the forwarded logs go nowhere. Enable TCP reception on the collector:
ON THE COLLECTOR: LISTEN ON TCP 514
printf 'module(load="imtcp")\ninput(type="imtcp" port="514")\n' | sudo tee /etc/rsyslog.d/10-listen.conf && sudo systemctl restart rsyslog && sudo ss -tlnp | grep ':514'Verify end to end: run logger -t labtest "forward check" on the client, then search for labtest in the collector logs.
Security note: bare TCP syslog is unencrypted and unauthenticated. On any untrusted network, use TLS (imtcp with certificates) or RELP with TLS instead of plain @@host:514.
Expected output:
What this teaches: logrotate -d dry-runs the config, showing which pattern matches and whether rotation is due, without touching any files.
How It Works
Modern Linux logging has two layers. journald captures stdout/stderr of every systemd service into a structured, indexed binary journal with fields like _SYSTEMD_UNIT and PRIORITY; journalctl queries that index, which is why filtering by unit, time, and priority is instant. rsyslog handles the traditional text-file side (/var/log/syslog) and simple network forwarding. how journal entries reach rsyslog depends on the distribution and local configuration: either journald forwards them to the local syslog socket (ForwardToSyslog=yes in journald.conf), or rsyslog pulls them from the journal directly with its imjournal input module. Upstream systemd defaults ForwardToSyslog to no, precisely because rsyslog and syslog-ng normally read the journal themselves, so check /etc/systemd/journald.conf and whether rsyslog loads imjournal to know which path your host uses. journald also ships its own network transport via systemd-journal-remote and systemd-journal-upload; rsyslog forwarding remains the simplest cross-platform path.
Centralized logging is a security control, not just convenience: an attacker who roots a host can tamper with local logs, but already-forwarded entries on the collector are out of reach. Forwarding over TCP (or TLS with certificates) plus local buffering covers the gap between event and delivery.
logrotate bounds disk usage with declarative policy: rotate weekly, keep 4, compress the old ones, and notify the daemon (postrotate reload) so it reopens files. Without rotation, one chatty service eventually fills the disk and takes the host down with it.
journald retention is separate from logrotate: write SystemMaxUse=500M under [Journal] in /etc/systemd/journald.conf.d/10-limits.conf, apply it with sudo systemctl restart systemd-journald, and trim on demand with sudo journalctl --vacuum-size=500M (vacuum needs root). logrotate never touches the binary journal.
Progress So Far
Testing & Validation
logger "test-event-tyf" && journalctl -n 1 | grep test-event-tyfYou should see the test event in the local journal (and on the collector if forwarding). If so, your logging pipeline is working.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| journalctl shows no logs for a unit | Service logs to a file instead of stdout, or Storage=none | Check journald.conf Storage setting and the unit's StandardOutput |
| Forwarded logs never arrive | Firewall, wrong port, or TCP vs UDP mismatch | Test with logger and tcpdump; @@ is TCP, @ is UDP |
| logrotate: error: state file | Concurrent runs or corrupt status file | Check /var/lib/logrotate/status and ensure one daily cron run |
| Logs stop after rotation | Daemon still writes to the renamed file handle | Add postrotate with systemctl reload/kill -HUP so it reopens logs |
| Journal consumes too much disk | SystemMaxUse unset on a busy host | Write SystemMaxUse=500M under [Journal] in /etc/systemd/journald.conf.d/10-limits.conf, run sudo systemctl restart systemd-journald, then trim with sudo journalctl --vacuum-size=500M |
| Timestamps confusing across hosts | Mixed timezones in text logs | Standardize on UTC via timedatectl and let the viewer localize |
Troubleshooting
- No logs on collector: confirm the collector listens on 514 and firewall allows it.
- Journal too large: cap with
SystemMaxUse=in journald.conf. - logrotate not running: it is triggered by a daily timer; test with
logrotate -f.
Extension Ideas
- Feed centralized logs into the Automated SOC Pipeline.
- Add structured logging (JSON) for easier parsing.
- Forward over TLS (RELP) for confidentiality and reliability.
Key Results
- Queried logs precisely by priority, time, and unit.
- Centralized logs to a collector for cross-host correlation.
- Bounded disk usage with rotation and retention.
- Verified the pipeline end to end with a test event.
Key Takeaways
- journalctl filters (unit, priority, since) make triage instant.
- Forward logs centrally; local logs are tamperable evidence.
- logrotate policy bounds disk: frequency, count, compression, postrotate.
- Dry-run with -d before trusting a new rotation config.
- UTC everywhere; localize at display time.
- Structured JSON logs query far better than free text once you scale.
- Rate limits such as RateLimitIntervalSec stop a noisy service from filling the disk.
Practice Challenges
Challenge 1: Breach timeline
Goal: using journalctl only, build a minute-by-minute timeline of SSH logins and failures for one day, exported to a file.
Hint: -o short-iso plus grep and sort.
Challenge 2: Forward and verify
Goal: forward a test host's auth logs to a collector, generate login events, and prove they arrived within seconds.
Hint: logger -p authpriv.info creates test events.