Incident Detection & Analysis

Intermediate Security
Splunklog analysisalert triageIOCsSIEM

Objective

Detect security incidents by analyzing logs, correlating events, identifying IOCs, and performing alert triage.

Tools & Technologies

  • Splunk
  • log analysis
  • alert triage
  • IOCs
  • SIEM

Key Commands

grep 'Failed password' /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn
journalctl -p warning --since '1 hour ago'
ss -tunp
last -n 30

Architecture Overview

sequenceDiagram participant LOG as Log Sources participant SIEM as SIEM/Splunk participant ANALYST as SOC Analyst participant TICKET as Ticket System LOG->>SIEM: Forward logs (syslog/agent) SIEM->>SIEM: Correlate & alert SIEM->>ANALYST: Alert notification ANALYST->>ANALYST: Triage alert (FP/TP?) Note over ANALYST: Check IOCs, context ANALYST->>TICKET: Create incident ticket ANALYST->>TICKET: Document findings

Step-by-Step Process

01
Log-Based Detection

Identify suspicious activity in system logs.

# Brute force detection
grep 'Failed password' /var/log/auth.log \
  | awk '{print $11}' | sort | uniq -c | sort -rn

# Successful logins from unusual IPs
grep 'Accepted' /var/log/auth.log | awk '{print $11}' | sort -u

# Privilege escalation
grep 'sudo' /var/log/auth.log | grep -v 'session opened'
02
Network-Based Detection

Identify suspicious network connections.

# Current connections
ss -tunp
# Listening services
ss -tunlp
# New outbound connections
netstat -antp | grep ESTABLISHED
# Unusual outbound ports
ss -tunp | awk '{print $5}' | cut -d: -f2 | sort | uniq -c | sort -rn
03
Correlate Events

Build a timeline of suspicious activity.

# Correlate auth events with network activity
# 1. Find when attacker logged in
grep 'Accepted password' /var/log/auth.log | tail -20
# 2. Find what they ran (if shell logging enabled)
grep '^2024-01-15 14:' /var/log/bash_history
# 3. Find network connections during that window
tcpdump -r capture.pcap -nn 'src host ATTACKER_IP'

Challenges & Solutions

  • High-volume alerts cause alert fatigue — tune thresholds after initial tuning
  • First responder should observe and document before taking action — avoid destroying evidence

Key Takeaways

  • Timeline is everything in IR — correlate events by timestamp across all sources
  • Attacker time zone may differ — convert all timestamps to UTC for analysis