Debugging a Production Incident with an Agent: What Changes, What Doesn't
In the articles on heap dumps and thread dumps, I insisted on one discipline: never conclude from a single measurement, always compare several dumps spread over time. Since I started debugging some of my incidents with an agent instead of reading dumps by hand, that discipline is exactly what hasn't moved. What moved is everything around it.
What an agent does better, and faster#
Reading a raw dump and correlating it with source code is exactly the kind
of pattern matching an agent is good at. Paste the "Found one Java-level
deadlock" block from the thread dump article, with repo access, and the
agent identifies the inconsistent lock acquisition order, cites the exact
line in Account.transfer, and proposes the fix (always lock the account
with the smaller ID first) without me having to do the dump-to-code
round trip myself.
| Task | Before, by hand | With an agent |
|---|---|---|
| Spotting the dominator in a MAT report | Walk the tree manually, sort by retained size | Paste the "Leak Suspects" report, the agent points straight to the responsible field and the path to GC roots |
| Correlating a stuck thread with code | Search the repo for the exact line from the stack trace | The agent already has repo access, connects stack trace to code in one pass |
| Writing the post-mortem | 30 to 45 minutes of writing after the fact | A first draft generated directly from the dumps and commands run during the incident |
That gain is real, but it's a gain in reading speed, not in diagnostic quality. That's where the distinction starts to matter.
What doesn't change: the discipline of evidence#
An agent asked to explain a dump produces a plausible-sounding story just as easily from a single snapshot as from three measurements spread over time. The difference between the two, a real memory leak signal versus a normal spike that drops back down after a full GC, doesn't jump out at a model any more than it does at a rushed human. That discipline has to be imposed from the outside: explicitly requiring several dumps before any conclusion, never leaving it to the model's judgment.
The principle is the same as the one in the article on auditing an architecture with an agent: an agent should never be the source of a fact, only the writer of a synthesis built from facts a deterministic tool produced. On an incident, that translates into a simple rule to enforce in the prompt or in a hook: every conclusion has to cite a specific line from a specific dump or log, never a claim with no pointer back to the raw evidence behind it.
What also doesn't change: production access#
The temptation, mid-incident, is to give the agent direct access to the server so it can generate and read the dumps itself. That's exactly the wrong moment to test the limits of unrestricted access: a production incident is never the place to discover that a misconfigured hook (see agent, harness, skills, hooks) lets the agent run a command you would never have approved with a clear head. What actually holds up is exporting the dumps and logs to a workspace the agent can read, and keeping any command that changes system state outside its reach during the incident itself. The diagram summarizes the whole loop, including the verification cycle and the access boundary:
A concrete example#
Take the cache that never stops growing, already described in the heap
dump article. An agent handed the "Leak Suspects" report identifies the
static HashMap with no expiration policy in seconds, and proposes the
Caffeine-based fix. What it doesn't do on its own is verify that the memory
floor (usage right after a full GC) is actually rising from one dump to
the next, rather than being an isolated spike tied to a one-off traffic
burst. That check remains something I have to ask for explicitly, with a
second dump taken later, before accepting the conclusion.
What this changes in practice#
The net gain is the time between the incident and a solid first hypothesis: what used to take twenty minutes of reading dumps now takes two minutes of synthesis. The two things that actually prevent a wrong post-mortem, repeated measurement over time and citing raw evidence instead of a plausible story, don't come automatically with the agent. They stay a discipline to design into the process, exactly as before, just now applied to a collaborator that reads faster than I ever could.