top of page

Debug Memory Leaks — Debugging Systematically, Part 6

Shift Quality Editorial
Jul 16
3 min read

Updated: Oct 4

Debugging Systematically · Part 6

A slow memory leak is one of the sneakiest bugs there is: everything works fine in testing, then the process gets OOM-killed at 3 a.m. after four days of uptime. Finding it means catching what the program forgot to let go of. This walks through debugging leaks across Python, Node, and the JVM — confirming it's really a leak, reproducing it, and using the right profiler to find what's holding on.


Memory usage climbs steadily. Eventually OOM. This tutorial walks the diagnosis.


Step 1: Confirm It's a Leak (10 min)


Not all growth is a leak. Some apps have legitimate caches, buffers, etc.


A leak is monotonic growth over time with steady workload:


Hour 1: 200 MB
Hour 2: 400 MB
Hour 3: 600 MB
Hour 4: 800 MB

vs. cache growth:


Hour 1: 200 MB
Hour 2: 400 MB
Hour 3: 500 MB
Hour 4: 500 MB  ← stable

Plot memory over time. Steady-state load + growing memory = leak.


Step 2: Reproduce in Isolation (10 min)


Best case: reproduce locally with constant load:


# Run a load test
ab -n 100000 -c 10 http://localhost:8000/endpoint &

# Watch memory
watch -n 5 'ps aux | grep "myapp" | head'

If memory grows: confirmed.


If you can't reproduce: skip to live production tools.


Step 3: Python: tracemalloc (15 min)


import tracemalloc

tracemalloc.start()

# Take a snapshot at start
snapshot1 = tracemalloc.take_snapshot()

# ... let the app run a while ...

snapshot2 = tracemalloc.take_snapshot()

top_stats = snapshot2.compare_to(snapshot1, "lineno")
for stat in top_stats[:10]:
    print(stat)

Output:


file.py:42 +100 MB

The line where memory grew most. Often points right at the leak.


Step 4: Python: objgraph (10 min)


import objgraph

# Show what types are most common
objgraph.show_most_common_types()

# Find references holding an object alive
objgraph.show_backrefs([leaky_object], filename="refs.png")

Visual: which references prevent garbage collection. Often the smoking gun.


Step 5: Node.js: --inspect + Chrome DevTools (15 min)


node --inspect myapp.js

Open Chrome → DevTools → Memory tab.


Take heap snapshots before and after. Compare. Look at "Comparison" view: what objects increased?


Or use clinic.js: clinic doctor -- node myapp.js.


Step 6: Java / JVM: jmap + jhat / VisualVM (15 min)


# Heap dump
jmap -dump:live,format=b,file=heap.bin <pid>

# Analyze
jhat heap.bin
# Or open in VisualVM, Eclipse MAT

MAT's "Leak Suspects" report is gold. Auto-detects common leak patterns.


Step 7: Look for Common Patterns (15 min)


Unbounded caches:


cache = {}  # never evicts

Use TTL or size-limited (e.g., functools.lru_cache(maxsize=1000)).


Event listener accumulation:


emitter.on("event", listener)  // never removed

emitter.removeListener when done.


Circular references (some GC handles; some don't):


a.ref = b
b.ref = a

For Python: usually GC'd. For some runtimes: leaks.


Long-lived closures:


function setupHandler() {
  const huge = loadHugeData();
  return () => smallThing(huge);  // huge stays in memory forever
}

Step 8: Profile in Production (10 min)


Light-weight profilers attach to running processes:


  • Python: py-spy, pyspy, austin

  • Node.js: clinic, 0x

  • JVM: JFR (Java Flight Recorder)

  • Go: built-in pprof


py-spy dump --pid 1234
py-spy top --pid 1234
py-spy record -o profile.svg --pid 1234

No restart needed. Production-safe.


Step 9: Continuous Profiling (10 min)


For ongoing visibility:


  • Pyroscope: continuous profiling for many languages

  • Datadog Profiling

  • Sentry Profiling

  • Go's pprof in production


Always-on. Spot leaks before they cause an outage.


Step 10: Restart vs. Fix (5 min)


Quick fix: restart on memory threshold (auto-recycle).


Real fix: find and fix the leak. Restarting is a workaround; the underlying bug still exists.


Both are legitimate at different stages:


  • Outage now: restart, recycle, mitigate

  • After: profile, find, fix


Don't get stuck only restarting.


What You Just Did


Memory leak diagnosis: confirm, reproduce, language-specific tools, common patterns, production profilers, continuous profiling, mitigation vs. fix. The full memory toolbox.


Common Failure Modes


Restart and forget. Leak is still there.


Assume cache = leak. Profile to distinguish.


One snapshot. Compare two to find growth.


Ignore in dev; debug in prod under load. Reproduce locally first.


Profile after restart. Memory has been freed; useless data.



Continue the Debugging Systematically path



Part of the Debugging Systematically learning path.

bottom of page