Linux Log Management with journald, rsyslog, and logrotate
You will query journald, forward logs to a central rsyslog server, and rotate files so disks never fill. By the end you will have searchable, retained, centralized logs, the foundation for monitoring and the SOC labs.
Learning Objectives
- Filter and follow logs with journalctl.
- Forward logs to a central server over rsyslog.
- Rotate and retain logs with logrotate.
- Time: ~3 hours · Difficulty: Intermediate · Prereqs: a server (and optionally a second host as collector).
Architecture Overview
graph LR
App[Apps + kernel] -->|journald| J[systemd-journald]
J -->|forward| R[rsyslog]
R -->|TCP/514| C[(Central log host)]
J -->|rotate| LR[logrotate retention]
Environment Setup
You will need: systemd-journald (default), rsyslog, and logrotate.
Before you begin: if forwarding, have a reachable collector listening on TCP/514.
Step-by-Step Execution
01
Query logs precisely with journalctl
Targeted filters turn a flood of entries into the few that matter for triage.
journalctl -p err -S "2026-06-17 00:00:00" -u ssh --no-pagerShows SSH errors since midnight at priority error or higher.
02
Forward logs to a central server
Centralized logs survive a compromised host and enable cross-host correlation.
echo '*.* @@192.168.1.60:514' | sudo tee /etc/rsyslog.d/90-forward.conf && sudo systemctl restart rsyslog03
Rotate logs with logrotate
# logrotate -d /etc/logrotate.d/rsyslog
rotating pattern: /var/log/syslog weekly (4 rotations)
considering log /var/log/syslog
log does not need rotating
Progress So Far
graph LR
A[01 Query journald] -->|done| B[02 Forward rsyslog]
B -->|done| C[03 Rotate logrotate]
style A fill:#1a4a1a,stroke:#00ff00,color:#fff
style B fill:#1a4a1a,stroke:#00ff00,color:#fff
style C fill:#1a4a1a,stroke:#00ff00,color:#fff
Testing & Validation
logger "test-event-tyf" && journalctl -n 1 | grep test-event-tyfYou should see the test event in the local journal (and on the collector if forwarding). If so, your logging pipeline is working.
Troubleshooting
- No logs on collector: confirm the collector listens on 514 and firewall allows it.
- Journal too large: cap with
SystemMaxUse=in journald.conf. - logrotate not running: it is triggered by a daily timer; test with
logrotate -f.
Extension Ideas
- Feed centralized logs into the Automated SOC Pipeline.
- Add structured logging (JSON) for easier parsing.
- Forward over TLS (RELP) for confidentiality and reliability.
Key Results
- Queried logs precisely by priority, time, and unit.
- Centralized logs to a collector for cross-host correlation.
- Bounded disk usage with rotation and retention.
- Verified the pipeline end to end with a test event.