Python Memory Usage Analyzer with tracemalloc and psutil
I built a memory profiler that snapshots allocations with tracemalloc, tracks process RSS with psutil, and inspects the garbage collector. Identifying the top allocators let me replace a list accumulation with a generator and cut peak memory substantially.
Objective & Context
Memory bloat is invisible until a script is OOM-killed. This lab measures rather than guesses, using tracemalloc to rank allocation sites and psutil to track real RSS, then applies streaming patterns to reduce footprint.
Learning Objectives
- Snapshot allocations with tracemalloc and rank top allocators.
- Track process RSS with psutil.
- Inspect garbage collector behaviour with gc.
- Identify accumulation patterns that waste memory.
- Refactor list accumulation into lazy generators.
Environment & Prerequisites
Prerequisites
Python 3.11withtracemallocandgc(standard library)psutilinstalled (pip install psutil)- A memory-heavy script and a representative input dataset to profile
- Assumed skill: functions, loops, and comprehensions
Step-by-Step Execution
1. Snapshot top allocators
import tracemalloc
tracemalloc.start()
data = [object() for _ in range(1_000_000)]
for stat in tracemalloc.take_snapshot().statistics("lineno")[:3]:
print(stat)Expected output:
<stdin>:4: size=23.3 MiB, count=1000001, average=24 B
(sample; your line numbers and byte counts will differ)What this teaches: statistics('lineno') attributes every allocation to a source line, so the report names the exact line to fix instead of a vague 'high memory'. The sample workload here is one million objects; swap in your own workload between start() and the snapshot to profile real code.
2. Track process RSS with psutil
python -c "import psutil,os; print(psutil.Process(os.getpid()).memory_info().rss//1024//1024,'MB')"Expected output:
(prints current RSS in MB, e.g. 214 MB)What this teaches: RSS is the ground truth from the OS; comparing it before and after a change proves the optimization worked outside the interpreter's accounting.
3. Refactor accumulation to a generator
python -c "g=(x*x for x in range(10**6)); print(sum(g))"Expected output:
333332833333500000What this teaches: The generator computes the same sum without materializing a million-element list; the result is identical while peak memory collapses. Compare with sum([x*x for x in range(10**6)]) under tracemalloc to see the peak difference directly.
How It Works
How It Works
tracemalloc hooks Python's allocator and records where each block was allocated. take_snapshot() freezes that accounting so statistics('lineno') can rank source lines by bytes held, turning 'the process is big' into 'line 42 holds 128 MiB'.
RSS (resident set size) is what the OS actually reserved for the process, including interpreter overhead and fragmentation that tracemalloc cannot see. Profiling with both gives the full picture: tracemalloc finds the Python-level culprit, RSS confirms the real-world impact.
Generators trade time for space: a generator expression yields one item at a time instead of building the whole list. When the consumer only needs each item once (sum, iteration, filtering), laziness removes the peak without changing the result.
result identical"]
Validation & Testing
Compare tracemalloc snapshots and RSS before and after the generator refactor on the same input. Pass criteria: the top allocator is identified, peak RSS drops measurably, and output is unchanged after the optimization.
Advanced: Troubleshooting
- Memory not freed: drop references and call
gc.collect(); watch for lingering globals. - Reference cycles: inspect with
gc.get_referrers; prefer weakrefs for caches. - Misleading RSS: the allocator may retain freed memory; trust tracemalloc deltas.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| tracemalloc shows nothing useful | tracing started after the allocations happened | call tracemalloc.start() before the workload, not after |
| RSS stays high after freeing objects | the allocator returned memory to its arenas but not to the OS | this is normal; watch for growth across repeated runs, not the absolute number |
| generator is slower than the list | the workload is CPU-bound or re-iterates the data | generators save memory, not time; materialize when you need multiple passes |
| compare snapshots shows negative noise | small allocations churn between snapshots | filter to the top-N statistics and focus on the dominant lines |
| gc.collect() changes nothing | the objects are still referenced, not garbage | use gc.get_referrers or objgraph-style tracing to find who holds them |
Key Results
- Identified the single top allocation site via tracemalloc ranking.
- Cut peak RSS by streaming with a generator instead of a list.
- Verified unchanged output after the memory optimization.
- Built a reusable profiling harness for future scripts.
Key Takeaways
- tracemalloc pinpoints the exact source lines behind memory growth.
- RSS from psutil is the OS-level ground truth for process size.
- Rank allocations by lineno to find the one line worth fixing.
- Generators eliminate materialization peaks when data is consumed once.
- Verify optimizations with before/after RSS, not just theory.
Practice Challenges
Challenge 1: Leak hunt
Goal: Write a script that appends to a global list in a loop, snapshot with tracemalloc, and identify the line from the report alone.
Hint: Take snapshots before and after the loop and compare them; the delta names the leak.
Challenge 2: Lazy pipeline
Goal: Convert a three-stage list pipeline (read, transform, filter) into chained generators and measure peak RSS on 1M rows.
Hint: Each stage becomes a generator expression feeding the next; sum() at the end drives the chain.