Investigating a JVM stall with a heap dump: capture, read, resolve
Before generating a heap dump, the first question to settle is often skipped: is this actually a memory problem? A component that "stalls" can just as easily be a deadlock between threads (no excessive memory, just threads waiting on each other) as a genuine memory drift. Those are two different diagnoses, with two different tools.
First, the right tool for the right symptom
| Symptom | Right tool | What it reveals |
|---|---|---|
| Threads stuck, CPU idle, no progress | Thread dump (jcmd <pid> Thread.print) | Every thread's state and which locks it's waiting on |
| Memory growing, GC firing more and more often, possibly an OutOfMemoryError | Heap dump | The heap's contents, object by object, with their references |
| CPU at 100% with no apparent stall | CPU profiling (async-profiler, JFR) | Where CPU time is actually being spent |
A heap dump on a deadlock shows nothing useful: memory isn't the problem, the threads are. That's the most common trap for anyone new to these tools: reaching for a heap dump by reflex on any "it stopped responding."
Generating a heap dump
Three common ways, depending on the context:
# On demand, on a running process (jcmd is the recommended tool today)
jcmd <pid> GC.heap_dump /path/heap.hprof
# Capture only live objects (excludes what the GC would reclaim anyway)
jcmd <pid> GC.heap_dump -all=false /path/heap-live.hprof
# Automatically at the moment of an OutOfMemoryError, no manual step needed
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/heap-oom.hprof
The HeapDumpOnOutOfMemoryError flag deserves to be enabled by default in
production: a dump taken after the fact, once the service has restarted,
no longer captures the state that caused the incident. The one moment this
dump has real value is exactly when the JVM realizes it can't continue.
Reading the dump: shallow size vs retained size
Opening a dump with Eclipse Memory Analyzer (MAT) shows a list of objects sortable by size, and that's exactly where misreading begins. Two metrics coexist, and confusing them leads straight to a wrong diagnosis:
| Metric | What it measures | Trap if ignored |
|---|---|---|
| Shallow size | The size of the object itself, without what it references | An empty HashMap has a tiny shallow size, even while retaining gigabytes of data through its entries |
| Retained size | The total memory freed if this object became unreachable (the object plus everything it exclusively retains) | This is the metric that actually reveals where the problem is |
Sorting by shallow size almost always surfaces generic types (char[],
HashMap$Node) without saying which specific one is the actual problem.
Sorting by retained size, via MAT's Dominator Tree, surfaces the root
object that's actually holding the memory, regardless of its own size.
A concrete case: the cache that never stops growing
MAT's "Leak Suspects" report often points to a single dominator responsible
for most of the heap. In a frequent case, that dominator is a static
HashMap used as an application cache, with no size limit or expiration
policy. The "Path to GC Roots" feature lets you trace the exact chain of
references keeping the garbage collector from freeing it: typically, a
static field on a Spring singleton, referenced from the classloader
itself, so never eligible for GC as long as the application runs.
// The trap: a cache that grows indefinitely
private static final Map<String, ExpensiveObject> cache = new HashMap<>();
// A direction for the fix: explicitly bound both size and lifetime
private static final Cache<String, ExpensiveObject> cache = Caffeine.newBuilder()
.maximumSize(10_000)
.expireAfterWrite(Duration.ofMinutes(30))
.build();
The HashMap isn't at fault by itself: it's the missing eviction policy
that turns a legitimate cache into a slow memory leak, invisible for weeks
until the number of distinct keys outgrows what the heap can absorb.
Don't confuse a leak with a normal spike
A single heap dump, taken at one point in time, says nothing about the trend: high memory usage right after a traffic spike can be perfectly normal if it drops back down after a full GC pass. The signal of a real leak isn't a full heap, it's a memory floor (usage right after a full GC) that keeps rising from one dump to the next, across several measurements spread over time, never a single isolated snapshot.
What a heap dump doesn't replace
A heap dump identifies where the memory is, not why the code accumulated it with that intent. Once the reference chain is found, the fix remains a matter of understanding the domain: was this cache really meant to keep everything, or did the author simply forget to add an expiration policy when writing it. The dump gives the exact location of the problem; it never gives the business justification for whether the data was supposed to stay there or not.