Log Management & Analysis

Objective

Navigate and analyze Linux system logs using journalctl and traditional log files, configure log rotation, and build alerting from log analysis.

Tools & Technologies

journalctl
rsyslog
logrotate
tail
/var/log/

Key Commands

journalctl -u service -n 100

tail -f /var/log/auth.log

grep 'Failed password' /var/log/auth.log | awk '{print $11}' | sort | uniq -c

logrotate -d /etc/logrotate.conf

Architecture Overview

graph TD APP[Application] -->|writes| SYSLOG[rsyslog/syslog-ng] APP2[Systemd Service] -->|via journald| JOURNAL[systemd journal] SYSLOG --> AUTH[/var/log/auth.log] SYSLOG --> SYS[/var/log/syslog] SYSLOG --> NGINX[/var/log/nginx/] JOURNAL --> JOURNAL_DB[/run/log/journal/\nbinary DB] AUTH --> ROTATE[logrotate] SYS --> ROTATE ROTATE --> COMPRESSED[*.gz archives] style JOURNAL_DB fill:#1a1a2e,stroke:#00d4ff,color:#e0e0e0 style ROTATE fill:#1a1a2e,stroke:#ffd700,color:#ffd700

Step-by-Step Process

Key Log Files

Know where different types of events are logged.

/var/log/syslog         # general system messages
/var/log/auth.log       # authentication (SSH, sudo)
/var/log/kern.log       # kernel messages
/var/log/nginx/         # web server
/var/log/apt/           # package installs

# Real-time monitoring
tail -f /var/log/auth.log
tail -f /var/log/syslog

journalctl

journalctl queries the systemd binary journal.

journalctl                        # all logs (oldest first)
journalctl -r                     # reverse (newest first)
journalctl -f                     # follow live
journalctl -u ssh -n 50           # 50 lines for ssh service
journalctl --since '2 hours ago'
journalctl --since '2024-01-01' --until '2024-01-02'
journalctl -p err                 # errors and above
journalctl _UID=1001              # specific user

Log Analysis One-Liners

Combine grep, awk, sort, and uniq to extract insights.

# Failed SSH attempts
grep 'Failed password' /var/log/auth.log | \
  awk '{print $11}' | sort | uniq -c | sort -rn | head

# Top IPs hitting nginx
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# HTTP 4xx/5xx errors
grep -E '" [45][0-9]{2} ' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Log Rotation

logrotate prevents logs from filling the disk.

# View config
cat /etc/logrotate.conf
ls /etc/logrotate.d/

# Example: /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    postrotate
        systemctl reload myapp
    endscript
}

# Test without executing
logrotate -d /etc/logrotate.conf

Challenges & Solutions

journalctl output is in UTC by default — use --utc or localtime config
logrotate postrotate script errors silently fail

Key Takeaways

Send critical logs to a remote syslog server for security — local logs can be tampered with
journald stores compressed binary logs — significantly smaller than plain text