top of page

Debug Memory Leaks — Debugging Systematically, Part 6

Shawn West
Jul 16
3 min read

Updated: Jul 28

Debugging Systematically · Part 6

A slow memory leak is one of the sneakiest bugs there is: everything works fine in testing, then the process gets OOM-killed at 3 a.m. after four days of uptime. Finding it means catching what the program forgot to let go of. This walks through debugging leaks across Python, Node, and the JVM — confirming it's really a leak, reproducing it, and using the right profiler to find what's holding on.

Memory usage climbs steadily. Eventually OOM. This tutorial walks the diagnosis.

Step 1: Confirm It's a Leak (10 min)

Not all growth is a leak. Some apps have legitimate caches, buffers, etc.

A leak is monotonic growth over time with steady workload:

Hour 1: 200 MB
Hour 2: 400 MB
Hour 3: 600 MB
Hour 4: 800 MB

vs. cache growth:

Hour 1: 200 MB
Hour 2: 400 MB
Hour 3: 500 MB
Hour 4: 500 MB  ← stable

Plot memory over time. Steady-state load + growing memory = leak.

Step 2: Reproduce in Isolation (10 min)

Best case: reproduce locally with constant load:

# Run a load test
ab -n 100000 -c 10 http://localhost:8000/endpoint &

# Watch memory
watch -n 5 'ps aux | grep "myapp" | head'

If memory grows: confirmed.

If you can't reproduce: skip to live production tools.

Step 3: Python: tracemalloc (15 min)

import tracemalloc

tracemalloc.start()

# Take a snapshot at start
snapshot1 = tracemalloc.take_snapshot()

# ... let the app run a while ...

snapshot2 = tracemalloc.take_snapshot()

top_stats = snapshot2.compare_to(snapshot1, "lineno")
for stat in top_stats[:10]:
    print(stat)

Output:

file.py:42 +100 MB

The line where memory grew most. Often points right at the leak.

Step 4: Python: objgraph (10 min)

import objgraph

# Show what types are most common
objgraph.show_most_common_types()

# Find references holding an object alive
objgraph.show_backrefs([leaky_object], filename="refs.png")

Visual: which references prevent garbage collection. Often the smoking gun.

Step 5: Node.js: --inspect + Chrome DevTools (15 min)

node --inspect myapp.js

Open Chrome → DevTools → Memory tab.

Take heap snapshots before and after. Compare. Look at "Comparison" view: what objects increased?

Or use clinic.js: clinic doctor -- node myapp.js.

Step 6: Java / JVM: jmap + jhat / VisualVM (15 min)

# Heap dump
jmap -dump:live,format=b,file=heap.bin <pid>

# Analyze
jhat heap.bin
# Or open in VisualVM, Eclipse MAT

MAT's "Leak Suspects" report is gold. Auto-detects common leak patterns.

Step 7: Look for Common Patterns (15 min)

Unbounded caches:

cache = {}  # never evicts

Use TTL or size-limited (e.g., functools.lru_cache(maxsize=1000)).

Event listener accumulation:

emitter.on("event", listener)  // never removed

emitter.removeListener when done.

Circular references (some GC handles; some don't):

a.ref = b
b.ref = a

For Python: usually GC'd. For some runtimes: leaks.

Long-lived closures:

function setupHandler() {
  const huge = loadHugeData();
  return () => smallThing(huge);  // huge stays in memory forever
}

Step 8: Profile in Production (10 min)

Light-weight profilers attach to running processes:

  • Python: py-spy, pyspy, austin

  • Node.js: clinic, 0x

  • JVM: JFR (Java Flight Recorder)

  • Go: built-in pprof

py-spy dump --pid 1234
py-spy top --pid 1234
py-spy record -o profile.svg --pid 1234

No restart needed. Production-safe.

Step 9: Continuous Profiling (10 min)

For ongoing visibility:

  • Pyroscope: continuous profiling for many languages

  • Datadog Profiling

  • Sentry Profiling

  • Go's pprof in production

Always-on. Spot leaks before they cause an outage.

Step 10: Restart vs. Fix (5 min)

Quick fix: restart on memory threshold (auto-recycle).

Real fix: find and fix the leak. Restarting is a workaround; the underlying bug still exists.

Both are legitimate at different stages:

  • Outage now: restart, recycle, mitigate

  • After: profile, find, fix

Don't get stuck only restarting.

What You Just Did

Memory leak diagnosis: confirm, reproduce, language-specific tools, common patterns, production profilers, continuous profiling, mitigation vs. fix. The full memory toolbox.

Common Failure Modes

Restart and forget. Leak is still there.

Assume cache = leak. Profile to distinguish.

One snapshot. Compare two to find growth.

Ignore in dev; debug in prod under load. Reproduce locally first.

Profile after restart. Memory has been freed; useless data.

Continue the Debugging Systematically path

Part of the Debugging Systematically learning path.

bottom of page