You will query journald, forward logs to a central rsyslog server, and rotate files so disks never fill. By the end you will have searchable, retained, centralized logs, the foundation for monitoring and the SOC labs.

Learning Objectives

  • Filter and follow logs with journalctl.
  • Forward logs to a central server over rsyslog.
  • Rotate and retain logs with logrotate.
  • Time: ~3 hours · Difficulty: Intermediate · Prereqs: a server (and optionally a second host as collector).

Architecture Overview

Environment Setup

You will need: systemd-journald (default), rsyslog, and logrotate.

Before you begin: if forwarding, have a reachable collector listening on TCP/514.

Prerequisites

  • A systemd-based Linux host with journald and rsyslog installed.
  • Root or sudo access for rsyslog config and logrotate.
  • A central log server address for the forwarding step (or a second VM).
  • A service generating logs (ssh used in the examples).

Step-by-Step Execution

The outputs on this page come from a re-run on 14 November 2026 in an Ubuntu 24.04 container (arm64). Sample files, users, and hosts were created for the run, so names, numbers, and timestamps reflect that run.

01
Query logs precisely with journalctl

Targeted filters turn a flood of entries into the few that matter for triage.

journalctl -p info -S "2026-06-17 00:00:00" -u ssh --no-pager
Shows SSH messages at info priority or higher since midnight. Failed SSH logins are recorded at info priority, so -p err would hide exactly the lines you are hunting.

Output:

Nov 14 17:32:04 srv01 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Nov 14 17:32:04 srv01 sshd[457]: Server listening on 0.0.0.0 port 22.
Nov 14 17:32:04 srv01 sshd[457]: Server listening on :: port 22.
Nov 14 17:32:04 srv01 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Nov 14 17:32:06 srv01 sshd[462]: Invalid user admin from 127.0.0.1 port 53748
Nov 14 17:32:06 srv01 sshd[462]: pam_unix(sshd:auth): check pass; user unknown
Nov 14 17:32:06 srv01 sshd[462]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Nov 14 17:32:07 srv01 sshd[462]: Failed password for invalid user admin from 127.0.0.1 port 53748 ssh2
Nov 14 17:32:09 srv01 sshd[462]: Connection closed by invalid user admin 127.0.0.1 port 53748 [preauth]

What this teaches: -p filters by priority (err and higher would miss these auth lines, which are info), -S sets the start time, -u scopes to one unit: three filters turn a flood into the exact lines triage needs. Match the priority to what the service actually logs.

02
Forward logs to a central server

Centralized logs survive a compromised host and enable cross-host correlation.

echo '*.* @@192.168.1.60:514' | sudo tee /etc/rsyslog.d/90-forward.conf && sudo systemctl restart rsyslog

Output:

*.* @@192.168.1.60:514
$ systemctl is-active rsyslog
active

What this teaches: A one-line rsyslog rule forwards everything over TCP (@@) to the collector; centralized logs survive a compromised or dead host.

The collector (192.168.1.60) must actually be listening, or the forwarded logs go nowhere. Enable TCP reception on the collector:

ON THE COLLECTOR: LISTEN ON TCP 514

printf 'module(load="imtcp")\ninput(type="imtcp" port="514")\n' | sudo tee /etc/rsyslog.d/10-listen.conf && sudo systemctl restart rsyslog && sudo ss -tlnp | grep ':514'
[ROOT REQUIRED] Run on 192.168.1.60; also open TCP 514 in the collector firewall.

Verify end to end: run logger -t labtest "forward check" on the client, then search for labtest in the collector logs.

Security note: bare TCP syslog is unencrypted and unauthenticated. On any untrusted network, use TLS (imtcp with certificates) or RELP with TLS instead of plain @@host:514.

03
Rotate logs with logrotate

Output:

# logrotate -d /etc/logrotate.d/rsyslog
warning: logrotate in debug mode does nothing except printing debug messages! Consider using verbose mode (-v) instead if this is not what you want. reading config file /etc/logrotate.d/rsyslog Reading state from file: /var/lib/logrotate/status state file /var/lib/logrotate/status does not exist Allocating hash table for state file, size 64 entries Handling 1 logs rotating pattern: /var/log/syslog /var/log/mail.log /var/log/kern.log ...

What this teaches: logrotate -d dry-runs the config, showing which pattern matches and whether rotation is due, without touching any files.

How It Works

Modern Linux logging has two layers. journald captures stdout/stderr of every systemd service into a structured, indexed binary journal with fields like _SYSTEMD_UNIT and PRIORITY; journalctl queries that index, which is why filtering by unit, time, and priority is instant. rsyslog handles the traditional text-file side (/var/log/syslog) and simple network forwarding. how journal entries reach rsyslog depends on the distribution and local configuration: either journald forwards them to the local syslog socket (ForwardToSyslog=yes in journald.conf), or rsyslog pulls them from the journal directly with its imjournal input module. Upstream systemd defaults ForwardToSyslog to no, precisely because rsyslog and syslog-ng normally read the journal themselves, so check /etc/systemd/journald.conf and whether rsyslog loads imjournal to know which path your host uses. journald also ships its own network transport via systemd-journal-remote and systemd-journal-upload; rsyslog forwarding remains the simplest cross-platform path.

Centralized logging is a security control, not just convenience: an attacker who roots a host can tamper with local logs, but already-forwarded entries on the collector are out of reach. Forwarding over TCP (or TLS with certificates) plus local buffering covers the gap between event and delivery.

logrotate bounds disk usage with declarative policy: rotate weekly, keep 4, compress the old ones, and notify the daemon (postrotate reload) so it reopens files. Without rotation, one chatty service eventually fills the disk and takes the host down with it.

journald retention is separate from logrotate: write SystemMaxUse=500M under [Journal] in /etc/systemd/journald.conf.d/10-limits.conf, apply it with sudo systemctl restart systemd-journald, and trim on demand with sudo journalctl --vacuum-size=500M (vacuum needs root). logrotate never touches the binary journal.

Progress So Far

Testing & Validation

logger "test-event-tyf" && journalctl -n 1 | grep test-event-tyf

You should see the test event in the local journal (and on the collector if forwarding). If so, your logging pipeline is working.

Troubleshooting

SymptomLikely causeFix
journalctl shows no logs for a unitService logs to a file instead of stdout, or Storage=noneCheck journald.conf Storage setting and the unit's StandardOutput
Forwarded logs never arriveFirewall, wrong port, or TCP vs UDP mismatchTest with logger and tcpdump; @@ is TCP, @ is UDP
logrotate: error: state fileConcurrent runs or corrupt status fileCheck /var/lib/logrotate/status and ensure one daily cron run
Logs stop after rotationDaemon still writes to the renamed file handleAdd postrotate with systemctl reload/kill -HUP so it reopens logs
Journal consumes too much diskSystemMaxUse unset on a busy hostWrite SystemMaxUse=500M under [Journal] in /etc/systemd/journald.conf.d/10-limits.conf, run sudo systemctl restart systemd-journald, then trim with sudo journalctl --vacuum-size=500M
Timestamps confusing across hostsMixed timezones in text logsStandardize on UTC via timedatectl and let the viewer localize
Troubleshooting
  • No logs on collector: confirm the collector listens on 514 and firewall allows it.
  • Journal too large: cap with SystemMaxUse= in journald.conf.
  • logrotate not running: it is triggered by a daily timer; test with logrotate -f.

Extension Ideas

  • Feed centralized logs into the Automated SOC Pipeline.
  • Add structured logging (JSON) for easier parsing.
  • Forward over TLS (RELP) for confidentiality and reliability.

Key Results

  • Queried logs precisely by priority, time, and unit.
  • Centralized logs to a collector for cross-host correlation.
  • Bounded disk usage with rotation and retention.
  • Verified the pipeline end to end with a test event.

Key Takeaways

  • journalctl filters (unit, priority, since) make triage instant.
  • Forward logs centrally; local logs are tamperable evidence.
  • logrotate policy bounds disk: frequency, count, compression, postrotate.
  • Dry-run with -d before trusting a new rotation config.
  • UTC everywhere; localize at display time.
  • Structured JSON logs query far better than free text once you scale.
  • Rate limits such as RateLimitIntervalSec stop a noisy service from filling the disk.

Practice Challenges

Challenge 1: Breach timeline

Goal: using journalctl only, build a minute-by-minute timeline of SSH logins and failures for one day, exported to a file.

Hint: -o short-iso plus grep and sort.

Challenge 2: Forward and verify

Goal: forward a test host's auth logs to a collector, generate login events, and prove they arrived within seconds.

Hint: logger -p authpriv.info creates test events.

Frequently Asked Questions

How do I fix 'journalctl shows no logs for a unit'?

Likely cause: Service logs to a file instead of stdout, or Storage=none. Fix: Check journald.conf Storage setting and the unit's StandardOutput.

How do I fix 'Forwarded logs never arrive'?

Likely cause: Firewall, wrong port, or TCP vs UDP mismatch. Fix: Test with logger and tcpdump; @@ is TCP, @ is UDP.

How do I fix 'logrotate: error: state file'?

Likely cause: Concurrent runs or corrupt status file. Fix: Check /var/lib/logrotate/status and ensure one daily cron run.

How do I fix 'Logs stop after rotation'?

Likely cause: Daemon still writes to the renamed file handle. Fix: Add postrotate with systemctl reload/kill -HUP so it reopens logs.

How do I fix 'Journal consumes too much disk'?

Likely cause: SystemMaxUse unset on a busy host. Fix: Write SystemMaxUse=500M under [Journal] in /etc/systemd/journald.conf.d/10-limits.conf, run sudo systemctl restart systemd-journald, then trim with sudo journalctl --vacuum-size=500M.

Cite This Page

Citation formats

Plain text

Taki Sadik, "Linux Log Management with journald, rsyslog, and logrotate", TYFSADIK, https://tyfsadik.org/blog/labs/linux-server/log-management (accessed 2026-10-22).

BibTeX

@misc{sadik-log-management,
  author  = {Sadik, MD. Taki Yasir Faraji},
  title   = {Linux Log Management with journald, rsyslog, and logrotate},
  year    = {2026},
  url     = {https://tyfsadik.org/blog/labs/linux-server/log-management},
  urldate = {2026-10-22}
}

APA

Sadik, M. T. Y. F. (2026). Linux Log Management with journald, rsyslog, and logrotate. TYFSADIK. https://tyfsadik.org/blog/labs/linux-server/log-management