Python Memory Usage Analyzer with tracemalloc and psutil

IntermediatePython~3 hours
tracemallocpsutilgcprofiling

I built a memory profiler that snapshots allocations with tracemalloc, tracks process RSS with psutil, and inspects the garbage collector. Identifying the top allocators let me replace a list accumulation with a generator and cut peak memory substantially.

Objective & Context

Memory bloat is invisible until a script is OOM-killed. This lab measures rather than guesses, using tracemalloc to rank allocation sites and psutil to track real RSS, then applies streaming patterns to reduce footprint.

Environment & Prerequisites

  • Python 3.11 with tracemalloc and gc (standard library); psutil.
  • A memory-heavy script to profile.
  • A representative input dataset.

Step-by-Step Execution

1. Snapshot top allocators

import tracemalloc
tracemalloc.start()
run_workload()
for stat in tracemalloc.take_snapshot().statistics("lineno")[:5]:
    print(stat)

2. Track process RSS with psutil

python -c "import psutil,os; print(psutil.Process(os.getpid()).memory_info().rss//1024//1024,'MB')"

3. Refactor accumulation to a generator

python -c "g=(x*x for x in range(10**6)); print(sum(g))"
data.py:42: size=128 MiB, count=1000000
333332833333500000

Validation & Testing

Compare tracemalloc snapshots and RSS before and after the generator refactor on the same input. Pass criteria: the top allocator is identified, peak RSS drops measurably, and output is unchanged after the optimization.

Advanced: Troubleshooting
  • Memory not freed: drop references and call gc.collect(); watch for lingering globals.
  • Reference cycles: inspect with gc.get_referrers; prefer weakrefs for caches.
  • Misleading RSS: the allocator may retain freed memory; trust tracemalloc deltas.

Key Results

  • Identified the single top allocation site via tracemalloc ranking.
  • Cut peak RSS by streaming with a generator instead of a list.
  • Verified unchanged output after the memory optimization.
  • Built a reusable profiling harness for future scripts.

Related Labs