Incident Response Procedures Built on the NIST SP 800-61 Lifecycle

I built documented incident response playbooks around the four-phase NIST SP 800-61 lifecycle and drove triage from Splunk correlation searches. Codifying the workflow brought mean time to detect below 15 minutes for high-fidelity alert classes in the lab's tabletop runs.

Most first-time responders do not fail on tooling, they fail on process. Under pressure, people skip evidence capture, contain the wrong host, or reboot a machine that still holds volatile memory. This lab exists to remove improvisation: every phase of the NIST SP 800-61 lifecycle is written down as a runbook with explicit inputs, decision points, commands, and exit criteria, so a tired analyst at 3 a.m. produces the same quality of response as a senior engineer at noon.

The scenario thread running through the steps below is a realistic one: an externally sourced password-spraying campaign succeeds against a single account, the attacker logs in over VPN, and begins staging data on an internal file server. Each step shows the exact search, command, or action, the output you should expect, and the mistake most people make at that point.

Objective & Context

Ad-hoc response loses time and evidence. This lab turns the NIST SP 800-61 phases, Preparation, Detection & Analysis, Containment/Eradication/Recovery, and Post-Incident, into repeatable runbooks with explicit decision points and evidence-handling rules.

Two attack patterns anchor the playbooks, chosen because they exercise both ends of the severity spectrum:

• T1078 Valid Accounts, anomalous logon correlation as a primary detection.
• T1486 Data Encrypted for Impact, ransomware playbook with isolation-first containment.

Valid-account abuse (T1078) is the detection challenge: the attacker is using legitimate credentials, so nothing trips a signature. Detection has to come from behavioral correlation, failure bursts followed by success, logons at unusual hours, or a source IP touching accounts it has never touched. Ransomware (T1486) is the opposite problem: detection is usually trivial (the victim calls you), but the containment decision, isolate now versus capture volatile evidence first, is time-critical and irreversible. Writing both playbooks forces you to think about severity-driven branching rather than a single linear flow.

The playbooks cover twelve incident types in total, phishing, malware detonation, credential abuse, ransomware, insider misuse, lost devices, web defacement, DDoS, data exposure, rogue devices, third-party compromise, and denial of access, but all twelve reduce to the same skeleton: detect, classify, decide evidence-versus-containment, execute, document, review. The scenario below walks that skeleton end to end for the credential-abuse case.

The loop back from Post-Incident Review to Preparation is the part teams skip and later regret. Every tabletop and every real incident in this lab ends with concrete updates, a tuned threshold, a new lookup column, an amended isolation procedure, committed back into the playbooks before the case is closed.

Environment & Prerequisites

(no output; this is a configuration file snippet, restart the forwarder to apply it)

host                lastTime     age
fs01.lab.local      1747523145   62
ws-20-15.lab.local  1747523139   68
dc01.lab.local      1747523147   60

Methodology

Why severity drives the path

Each playbook branches on severity: SEV1 (active compromise) triggers immediate isolation, while SEV3 follows monitored analysis. This prevents over-reacting to noise and under-reacting to real intrusions.

The branching matters because the two failure modes are asymmetric in cost. Isolating a production file server on a false positive costs the business real money and burns the incident team's credibility; not isolating during a real ransomware detonation can cost the entire network. The runbooks resolve this with an explicit evidence-versus-containment rule: for SEV1 on ransomware-class incidents, isolate first and accept the volatile-evidence loss; for credential-abuse incidents where the host is stable, capture volatile data (memory, active connections, logged-on sessions) before pulling the network, because that evidence is often the only record of what the attacker actually did.

Communications: who needs to know, and when

SEV1 opens an incident bridge within 15 minutes of classification and posts status updates on a fixed cadence, every 30 minutes for SEV1, hourly for SEV2, so leadership never has to ask "what is happening," which is the single biggest time sink in real incidents. Crucially, comms for a suspected account-compromise incident run out-of-band (a pre-registered Signal group and personal phone numbers), because announcing the response over corporate email or chat tells an attacker who is watching those channels exactly what you know.

[IR-2026-0142 | SEV1 | Update 2 | 03:45 UTC]
Situation:  Spray attack from 203.0.113.45; 1 success (j.chen) on fs01.
Actions:    Volatile evidence captured 03:41; host isolated 03:44.
Next:       Credential rotation + persistence sweep; next update 04:15.
Impact:     File services unaffected; no confirmed exfiltration yet.

Evidence handling and chain of custody

Every artifact collected during an incident follows the same custody rules: a unique evidence ID (CASE-HOST-SEQ), the collector's name, UTC timestamps, the tool and version used, and a SHA-256 hash taken at collection time. The custody form itself is a template, filled in per artifact, and it travels with the evidence:

Evidence ID:   IR-2026-0142-fs01-003
Description:   Volatile memory image (AVML 0.14.0)
Collected by:  T. Sadik            UTC: 2026-06-17 03:41:07
Source host:   fs01.lab.local (192.168.20.15)
SHA-256:       9f2c...a41b
Custody log:   03:41 collected -> 03:55 copied to analysis share
               -> 04:02 hash re-verified (match)

The second methodological choice is that every decision point produces an artifact. Classification produces a severity tag on the case; containment produces a timestamped isolation record; eradication produces a list of removed persistence mechanisms. If a step did not produce an artifact, it did not happen, this is what makes the post-incident review honest and the MTTD/MTTR metrics measurable rather than vibes.

Step-by-Step Execution

1. Detection: a Splunk correlation search

The detection goal is a burst of authentication failures from one source, the fingerprint of password spraying or brute force. The naive version of this search counts failures per source IP and fires above a fixed threshold:

Expected output:

src_ip          count
203.0.113.45    87
198.51.100.23   31

That works, but it has a blind spot: a low-and-slow spray of 20 failures over six hours is very different from 20 failures in one minute. The production version buckets events into a 15-minute window, tracks distinct targeted accounts (spraying hits many accounts once; brute force hammers one account many times), and appends a success check, one success after a failure burst is the signal that matters most:

Expected output:

src_ip          fails  users_targeted  successes
203.0.113.45    312    47              1

Save this as a scheduled search (cron */5 * * * *, 15-minute window) with a trigger action that opens a case and pages on-call only when successes > 0; failure-only bursts go to a review queue. That single condition is what keeps the pager quiet at night while still catching the spray that worked. The saved search stanza in savedsearches.conf looks like this:

[Auth - Spray Burst with Success]
search = index=auth earliest=-15m | stats ... (as above)
cron_schedule = */5 * * * *
dispatch.earliest_time = -15m
alert_condition = search successes > 0
alert.suppress = 1
alert.suppress.period = 3600
alert.suppress.fields = src_ip

Throttling (alert.suppress) per source IP prevents one noisy incident from paging fifty times; one page per source per hour is enough, the case accumulates the rest. A baseline report run weekly (stats avg(fails) max(fails) by src_ip over 30 days) is how the threshold of 20 was chosen, it sits at roughly three standard deviations above the noisiest legitimate source in this environment.

Common mistake: alerting on the raw threshold without a time window, so a legitimate service account with a cached stale password pages the team every morning. Verify: run the search manually over the last 24 hours, you should see your known noise sources (they belong in a suppression list) and nothing unexpected.

What this teaches: detection is a query plus a threshold; the correlation search turns log noise into a countable signal.

2. Triage: enrich and classify severity

Detection tells you something fired; triage decides what it is. The enrichment pass answers three questions in under five minutes: Is the source known-bad? (threat intel lookup), What did it touch? (account and host inventory lookup), and Did anything succeed? (the success event itself, pivoting to the destination host and session).

Expected output:

_time                src_ip         threat_match
2026-09-29 12:01:04  203.0.113.45   yes

In practice the triage search is a single SPL pipeline that joins all three lookups at once, so the analyst gets one row per incident rather than three separate queries:

Expected output:

src_ip         dest            users  accounts            reputation  criticality
203.0.113.45   fs01.lab.local  47     {..., j.chen, ...}  malicious   high

Before classifying, rule out the two classic false positives: the source being one of your own VPN egress IPs (users mistype passwords in bursts, especially on Monday mornings) and a scheduled service with a stale credential. Both are handled by a suppression lookup of known egress ranges and service accounts, checked before escalation, not after waking up the on-call.

The final pivot inspects the successful session itself, which account, from where, logon type. Logon type 10 means RemoteInteractive (typically RDP); it is not VPN by itself, here the Workstation_Name of VPN-GW01 corroborates the remote-access path, and that corroboration is what makes it the smoking gun for hands-on activity rather than the type number alone:

_time                Account_Name  Workstation  Logon_Type  Logon_Process
2026-06-17 03:12:44  j.chen        VPN-GW01     10          User32

Here the source is flagged malicious by threat intel and it reached a high-criticality file server with a successful remote-interactive logon, that is textbook T1078 Valid Accounts, classified SEV1. Had the source been an internal scanner with zero successes, this would be SEV3 and would follow the monitored-analysis branch instead. The classification, its justification, and the ATT&CK mapping go into the case record immediately; the playbook you execute next is chosen by this tag.

Common mistake: classifying on the alert volume instead of the outcome, 312 failures feels worse than it is; one success is what makes it SEV1. Verify: pull the raw success event and confirm the logon type and destination yourself before escalating; never classify on an aggregate row alone.

What this teaches: triage converts a detection into a severity decision with business context attached.

3. Evidence capture: volatile data before the network pull

For a credential-abuse SEV1 on a host that is not actively detonating ransomware, the runbook's evidence-first rule applies: capture what disappears when the power or network drops. Order of volatility (RFC 3227) puts memory, network connections, and logged-on sessions first, disk artifacts survive a reboot, memory does not.

Every artifact is hashed at capture time and logged in the chain-of-custody form with collector, timestamp, and tool version, this is what makes the evidence defensible later:

Expected output:

9f2c...a41b  fs01_20260617.mem
1c77...09d2  fs01_netstat.txt
b4e0...5f88  fs01_sessions.txt

Hashes above are truncated for readability; the real SHA256SUMS.txt holds full 64-hex digests. Analysis happens on a copy, never the original. In the memory image from this scenario, a quick strings fs01_20260617.mem | grep -i smbclient pass (and a proper Volatility run later) surfaced an smbclient session to a staging share and a half-finished tar of the finance directory, the difference between "account compromised" and "exfiltration in progress," which changes the legal notification path.

Common mistake: running the capture tools after isolation on a host whose IR tooling phones home or writes to a network share, you have just cut off your own evidence path. Verify: sha256sum -c SHA256SUMS.txt re-run on the analysis copy; the memory image size should roughly match installed RAM (AVML images are compressed, so somewhat smaller is normal).

What this teaches: volatile evidence dies at containment, so capture order follows volatility.

4. Containment: isolate the host

Containment for this incident means network isolation that preserves the host for forensics, block everything except the IR subnet, so analysts retain reachability but the attacker loses their session and any exfiltration path. On the Linux file server this is an nftables egress rule. This requires root, the nftables package installed, and an existing inet filter table with an output chain (create one first with sudo nft add table inet filter && sudo nft add chain inet filter output if the lab host has none):

Expected output:

(no output on success; the case record "[SEV1] host 192.168.20.15 isolated; case IR-2026-0142 opened" is opened separately by the analyst)

Two details matter here. First, the rule allows the IR subnet (192.168.10.0/24) rather than a hard down, a total ip link set down is faster to type but leaves you with a host you can no longer examine remotely. Second, for ransomware-class incidents the runbook inverts this: switch-port shutdown or VLAN quarantine executed by the network team, because a software rule on a compromised host cannot be trusted once the attacker has root.

The playbook lists the containment options in order of trustworthiness, from most to least: switch-port shutdown (nothing on the host can override it), EDR-initiated network isolation (vendor agent, tamper-resistant but needs the agent alive), then host-based firewall rules like the nftables example, fastest, but only trustworthy while you still trust the kernel on that box.

table inet filter {
    chain output {
        type filter hook output priority filter; policy accept;
        ip daddr != 192.168.10.0/24 drop
    }
}
--- 203.0.113.45 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss

Keep the rollback ready before you apply the rule, containment you cannot undo quickly is an outage waiting for a change window. Rule handles are assigned by the kernel, so capture the real one at creation time (printed above) rather than guessing:

Expected output:

(no output on success)

The isolation record, timestamp, method, who executed it, verification output, is attached to the case. Common mistake: adding the drop rule after a broad accept rule that matches first; nftables evaluates top-down, so a misordered rule silently does nothing. Verify: always test with real traffic from the isolated host, not just nft list output.

What this teaches: isolation stops the bleeding while preserving the host for analysis, unlike a wipe.

5. Eradication & recovery: close the door, then rebuild trust

Eradication answers "how did they get in and how do we make sure they cannot walk back through the same door." For this scenario that means three concrete actions: rotate the compromised credential (and any account it could reach), kill the attacker's persistence, and patch or reconfigure the exposed path, here, enforcing MFA on the VPN that allowed password-only logon.

Expected output:

(no output on success)

On the Linux side the persistence sweep covers the usual hiding places, cron, SSH keys, and unexpected accounts:

# Splunk confirmation that the account goes quiet after rotation
index=wineventlog user=j.chen action=success earliest=-1h
# result: 0 events

Recovery is deliberately boring: restore the host to the production VLAN in stages (monitoring-only first, full service after 24 quiet hours), and only after the root cause is closed. If anything needed restoring from backup, the backup is verified before the restore, a clean-room mount and hash check, because restoring yesterday's backup of an already-compromised host just re-installs the attacker. Bringing a host back before the VPN gap is fixed just schedules incident number two.

Common mistake: resetting only the one account that fired the alert. The spray touched 47 accounts; any of them with a weak or reused password is the next incident. Verify: the post-rotation search returns zero successful logons, and the correlation search from step 1 shows no further bursts from any source.

What this teaches: eradication removes the attacker's footholds; recovery rebuilds trust in the system.

6. Post-incident: timeline, metrics, and playbook updates

The final phase converts the incident into institutional memory. Build the definitive timeline from the same data you triaged with, first failure, success, containment, eradication, recovery, and compute the two numbers the program is judged on:

Expected output:

MTTD: 11 min  (first spray event -> alert fired)
MTTC: 26 min  (alert fired -> isolation verified)
MTTR: 4 h 10 m (alert -> host back in full service)

The written report follows a fixed skeleton so reviews are comparable across incidents and nobody invents a format while tired:

• Executive summary, three sentences a director can read: what happened, what it cost, what changed.
• Timeline, every event and action with UTC timestamps, from first spray to case close.
• Root cause, the VPN's password-only logon path, not "a user had a weak password."
• Scope, the full list of 47 sprayed accounts, every host the session touched, data accessed.
• What worked / what failed, detection fired in 11 minutes; comms bridge took 25.
• Action items, each with a ticket number, owner, and due date.

The lessons-learned review (within five business days, blameless, with everyone who touched the case) produces the artifacts that feed back into Preparation: in this incident, a VPN-MFA gap ticket, a suppression-list entry for the scanner subnet, and a playbook amendment making the "failure burst + any success = page" condition explicit. The case only closes when those items are ticketed and owned.

Common mistake: treating the review as a formality and filing the report without updates, the loop back to Preparation in the lifecycle diagram above then does nothing. Verify: every action item has a ticket number and an owner in the report's closing section.

What this teaches: the post-incident review is where the organization actually learns; metrics prove it.

Validation & Testing

Run a tabletop using a simulated brute-force-then-lateral-movement scenario and time each phase. Pass criteria: detection under 15 minutes, containment decision documented, and a complete post-incident report with lessons learned.

The exercise is run with injects on a timer: the facilitator plants the attack telemetry (a scripted spray from the Kali box against the lab domain), starts the clock, and hands the team nothing but the alert. The team must execute steps 1 to 6 above against live data. Timings are captured from the case log, not self-reported, the alert timestamp, the evidence hash timestamps, and the isolation record are all already in the artifacts, which is the payoff of the artifact-per-decision rule. A red-team twist worth including once the basics are solid: one of the 47 sprayed accounts is a honeypot with a deliberately weak password, which tests whether the team scopes beyond the first compromised account in step 5.

Scoring rubric used across the tabletops:

• Detection < 15 min from first spray event to acknowledged alert.
• Correct severity assigned with a written justification citing the success event.
• Evidence captured before isolation, with hashes matching at verification.
• Containment verified by traffic test, not assumed from rule listing.
• Post-incident report complete with ticketed action items.

Three consecutive passing runs moved the playbooks from draft to operational status; the fourth run introduced the honeypot twist and caught a scoping miss, which is exactly what the exercise is for.

Troubleshooting

Key Results

• Reduced mean time to detect (MTTD) to under 15 minutes for priority alert classes.
• Documented 4 severity-branched playbooks covering 12 common incident types.
• Achieved 100% chain-of-custody completion across tabletop exercises.
• Cut average triage time by roughly 50% with pre-built enrichment lookups.

Key Takeaways

• Write the playbook before the incident: decision points, commands, and exit criteria belong on paper, not in someone's head.
• Detect the outcome, not the volume, "failure burst plus any success" beats a raw failure threshold for both fidelity and pager sanity.
• Let severity drive the path: isolate-first for destructive attacks, evidence-first for credential abuse, monitored analysis for the unconfirmed.
• Containment is not done until you have verified it with live traffic from the isolated host.
• Every step produces an artifact (severity tag, hashes, isolation record, timeline); artifacts are what make metrics and reviews honest.
• The lifecycle is a loop, an incident that does not update the playbooks is a lesson paid for and never collected.

Practice Challenges