Agent Postmortems Without Blame: A Format That Surfaces Causes And Fixes Without Shaming
An adapted SRE postmortem template for agent failures. Five sections: timeline, root cause, contributing factors, fix, prevention. No blame, no shame, durable learning.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
Most agent postmortems fail in one of two ways. They identify the agent as the cause and stop, which produces no learning. Or they devolve into blame of the humans who built or operated the agent, which produces defensiveness and fewer postmortems next time. Both failures share a root: the lack of a format that names causes structurally. This post adapts the SRE blameless postmortem template for agent failures. Five sections: timeline, root cause in three layers, contributing factors, fix in three layers, prevention through closed feedback loops. The template is filled out by the incident commander within seven days, reviewed by the agent owner and a peer, and published to a defined audience. The output is durable institutional learning, not embarrassment. Use it for every incident. The cumulative library becomes the institutional memory.
Intro: The Postmortem That Made Things Worse
Last quarter your team had an agent incident. The agent was a content moderator. It had been pulling unrelated user reports into a single decision and producing actions that did not match any single report. The pattern lasted nine days before the engineering lead noticed it during a routine review. The damage was contained but real: a few dozen users had been wrongly flagged.
You ran a postmortem. The postmortem was Friday afternoon. Twelve people in a conference room, two more on video. The format was open discussion. The first hour was timeline reconstruction, which surfaced that the agent's prompt template had been changed on day zero by an engineer who was now in the room and visibly uncomfortable. The second hour was root cause discussion, which kept circling back to the prompt change. By the end of the second hour, three people had pointed out, with varying degrees of tact, that the prompt change had been deployed without a corresponding update to the evaluation suite. The engineer who made the change took ownership in a way that felt to the room like an apology.
The meeting ended with action items. The action items were specific to the prompt change. Add an evaluation case for the failure mode. Require evaluation suite updates with prompt changes. Review the deploy pipeline. The action items shipped over the next two weeks. The team felt like they had done a thorough postmortem.
A quarter later, a different agent had a different incident. The pattern was different in surface and identical in structure. A configuration change had been deployed without a corresponding update to the evaluation suite. The change introduced a regression. The regression went undetected for days. The engineer who made the configuration change had not been in the room for the previous postmortem. They had read the action items but had not understood the deeper pattern, because the previous postmortem had focused on the specific prompt change and not on the structural condition that allowed any change to ship without a corresponding evaluation update.
Worse, the engineer in the second incident had heard, secondhand, that the previous postmortem had been uncomfortable for the engineer involved. They became defensive. The second postmortem was harder to run because the participants were now wary of being the next person whose name was associated with an incident.
This is the failure mode of unstructured postmortems. They produce action items that are too specific to the proximate cause and too vague about the systemic cause. They produce social dynamics that discourage the next round of openness. The cumulative effect is that the team has more incidents, fewer postmortems, and a culture that treats postmortems as something to survive rather than something that produces durable institutional learning.
The fix is a structured format. Five sections, a defined sequence, blameless framing, and a focus on systemic causes that transfer across incidents. This post is the format. Use it. The friction of structure is small. The benefit of durable learning compounds.
Why Agent Postmortems Need Their Own Template
The SRE community has decades of experience with blameless postmortems for service incidents. The principles transfer to agents but the surface area does not. Service postmortems focus on availability, latency, and correctness of code that humans wrote. Agent postmortems must additionally address the agent's pact, the audit trail of the agent's decisions, the evaluation suite that was supposed to catch the failure, the bond that may be slashed, and the disputes that may follow. The template needs to make all of this explicit so the postmortem author does not omit them under deadline pressure.
Agents also fail differently from services. A service that fails usually fails by stopping. An agent that fails usually fails by continuing to act, but acting in a way that violates the pact. The continuation is the hard part. The postmortem must reconstruct not just what the agent did but why it kept doing it, why no detector caught the deviation, why no escalation was triggered, why the human in the loop did not intervene. Each of these is a separate question with its own causes. The template enforces that all of them are asked.
Finally, agent failures touch counterparties differently from service failures. A service outage is communicated to customers. An agent failure is communicated to the customers it acted on, the counterparties whose trust oracle queries returned a now-wrong score, the platform that signed the pact, and possibly the dispute mechanism. The postmortem must support all of these communications. The template builds them in.
The template that follows is designed for agent incidents specifically. It can be used for adjacent failures, like prompt regressions or evaluation harness bugs, but it shines for full agent incidents where the pact, the bond, the audit trail, and the counterparty are all in play.
Section 1: Timeline
The timeline is the chronological reconstruction of the incident. It is the foundation for every other section. A timeline that is incomplete or contradictory produces root causes and fixes that do not match the actual events. The timeline is also the section where blamelessness is most easily violated. Names attached to timestamps invite reading the timeline as a story about people. The discipline is to attach roles to timestamps when possible and names only when the role is ambiguous.
The timeline starts before the incident is detected. The earliest entry is the change that introduced the failure condition, even if no one knew at the time that it was a failure condition. The latest entry is the closure of the immediate response. Everything in between is recorded with timestamps from authoritative sources: the audit trail, the deployment log, the incident channel, the dispute filings, the on-chain settlement records.
Each entry has a timestamp, a description, and a source. Entries that are reconstructed from imperfect memory are marked as such. The marking matters because it tells future readers which entries are evidentiary and which are interpretive. Mixing the two without distinction is how postmortems start to feel like fan fiction.
The timeline includes the moment of detection, the moment of acknowledgement, each containment action with its timestamp and the lever pulled, the snapshot moments, the communication updates internal and external, the hypothesis formation moments, the root cause confirmation moments, and the fix deployment moments. For incidents that produce a dispute, the timeline includes the dispute filing, the jury resolution, and the on-chain settlement. For incidents that produce a slashing, the timeline includes the slashing transaction.
A good timeline is dense in the first hour and the first day, sparse in the longer tail. The density reflects where the consequential decisions were made. The sparsity reflects where the response was running on the procedures established earlier. Either pattern is fine. What is not fine is a timeline that is uniformly sparse, because that means the response was unobserved or undocumented, which is the silent failure mode that makes future incidents harder to handle.
A second discipline is to include the non-events. Times when an alert should have fired but did not. Times when a detector was expected to catch something and did not. Times when a human was expected to be paged and was not. The non-events are the negative space of the timeline and they often contain the most important learning. Without them, the postmortem reads like a story of heroic response. With them, it reads like an honest account of where the system fell short.
The timeline section should be tight. A page or two. If it is longer, the timeline is probably mixing in interpretation that belongs in later sections. Pull the interpretation out. The timeline is what happened, not why.
Section 2: Root Cause In Three Layers
Root cause is the section where most postmortems fail. They name a single cause, usually proximate, and stop. The single-cause framing is comforting because it produces a single fix and feels complete. It is also misleading because real incidents almost always have layered causes, and naming only the proximate one produces fixes that prevent the specific incident and not the structural class.
The template requires three layers, named explicitly. Proximate cause. Contributing causes. Systemic cause.
Proximate cause. The specific sequence of inputs and decisions that produced the observed behavior. For the content moderator example: the agent's prompt template was changed on day zero in a way that allowed it to combine multiple report contexts when generating a moderation decision. The combination produced decisions that did not correspond to any single report.
Contributing causes. The conditions that made the proximate cause possible. There are usually multiple contributing causes and they cluster into the three categories the acceptance test exposes: pact under-specification, tool over-permission, evaluation under-coverage. For the content moderator: the pact did not address how the agent should handle multiple simultaneous reports about the same user. The tool that fetched report context allowed multiple reports to be fetched in a single call. The evaluation suite did not include cases where multiple reports were available for the same user.
Each contributing cause is named with evidence. Pact under-specification is shown with a quote of the pact and a mapping to the behavior it failed to constrain. Tool over-permission is shown with the tool definition and the input that should have been rejected. Evaluation under-coverage is shown with the suite and the missing case. Evidence makes the contributing causes hard to dismiss. Without evidence they read as opinion.
Systemic cause. The property of the operations practice that allowed the contributing causes to coexist for a production agent. For the content moderator: the prompt template change pipeline did not require an evaluation suite update as a precondition for deployment. The pipeline merged the prompt change with no signal that the evaluation suite needed to be extended.
The systemic cause is the highest-leverage section. It transfers across incidents. The fix to a systemic cause prevents not just the specific incident but the class of incidents. Postmortems that name systemic causes produce structural improvements. Postmortems that stop at proximate cause produce point patches.
A test for whether the three layers are real: imagine the postmortem reader is the operator of a different agent, with a different pact, in a different domain. Would they learn something useful from your postmortem? If the answer is yes, the layers are real. If the answer is no, the postmortem is too local. Rewrite.
Section 3: Contributing Factors
Contributing factors are the human and process conditions that allowed the technical conditions to exist. The distinction from contributing causes is subtle but important. Contributing causes are technical conditions. Contributing factors are organizational conditions. Both must be named for the postmortem to be complete.
The contributing factors section is where blamelessness is most tested. The honest account often involves naming process gaps, ownership ambiguities, time pressure, communication failures, and tooling limitations. Each of these has people associated with it. The discipline is to describe the conditions, not the people. The condition is what is fixable. The person is what gets defensive.
A well-written contributing factors section reads like a sociologist's analysis. It names the team's relationship to the work, the incentives that shaped the decisions, the information that was or was not available at decision time, and the structural pressures that made the right thing hard. It does not name individuals as failures. It names the system as imperfect.
For the content moderator example, contributing factors might include: the prompt template change was reviewed by a single engineer because the team's review practice did not require a second reviewer for prompt changes; the engineer making the change did not know that the evaluation suite needed to be extended because the connection between prompt changes and evaluation updates was not documented; the change was deployed during a sprint focused on a customer launch, which created time pressure that made deeper review feel optional; the team had been understaffed for two months due to attrition that had not yet been backfilled, which meant the engineer who would normally have caught the issue was not available.
Notice how each factor names a condition that can be addressed. The review practice can be updated. The connection between prompt and evaluation can be documented. The sprint pressure can be rebalanced. The staffing can be planned. None of these is a failure of an individual; all of them are properties of how the team works that the team can choose to change.
A postmortem that names contributing factors well becomes an organizational artifact. Future leaders read it and understand not just what went wrong but how the team was structured at the time. This understanding compounds across incidents and forms the basis for organizational improvement that goes beyond technical fixes.
Section 4: Fix In Three Layers
The fix section mirrors the root cause section. Three layers: proximate fix, contributing fix, systemic fix. Each fix has an owner, a target date, and a verification step.
Proximate fix. Prevents the specific path that triggered this incident. For the content moderator: update the prompt template to explicitly handle multiple reports as separate contexts. Owner named. Target date specified. Verification: replay the snapshot from the incident with the new template and confirm the agent makes per-report decisions.
Contributing fix. Closes the under-specification, over-permission, or under-coverage that made the path possible. For the content moderator: update the pact to address multiple-report scenarios, restrict the report-fetch tool to single-report calls unless the pact explicitly authorizes multiple, extend the evaluation suite with multiple-report cases. Each is an owner, a date, a verification.
Systemic fix. Creates the feedback loop whose absence allowed the contributing causes to coexist. For the content moderator: require, in the deploy pipeline, that any prompt template change include either an evaluation suite update or an explicit attestation that no update is needed, signed by a second reviewer. The pipeline change is the systemic fix. Verification: after the change, attempt to deploy a prompt change without an evaluation update and confirm the pipeline rejects.
The systemic fix is the most important and the most likely to be deferred. Teams under deadline pressure ship the proximate fix, mostly ship the contributing fix, and let the systemic fix slip. The slip is rational at the level of any single incident and irrational at the level of the cumulative incident pattern. The template requires the systemic fix to be named with an owner and a date even if the date is later than the others. The naming is what keeps the fix alive across the months it may take.
Each fix is verified before the postmortem is closed. Verification is replay against the incident snapshot, plus a forward test against the new condition. Without verification, the fix is asserted rather than confirmed, and asserted fixes have a way of not actually fixing what they claim. The verification step is the discipline that converts intent to evidence.
The fix section also names what is not being fixed and why. Every postmortem produces a list of plausible improvements. Most are not the highest-leverage thing to do next. Naming the unfixed items, with reasons, prevents them from becoming a backlog of zombie commitments. The discipline of saying not now and writing down why is one of the most undervalued habits in operations.
Section 5: Prevention Through Closed Feedback Loops
Prevention is the section that distinguishes a postmortem from a bug report. A bug report names what happened and what was fixed. A postmortem names what will prevent the class of incident from recurring. Prevention is not a fix; it is a feedback loop. The loop senses the conditions that produced the incident and triggers a response before they can produce another one.
For the content moderator, prevention might include: a continuous evaluation that includes scenarios where the agent's tool returns multiple results; a detector on the agent's outputs that flags decisions where the input contains more than one report context; a quarterly review of the pact against the actual production input distribution; a calibration of the evaluation suite against the previous quarter's anomalies. Each of these is a loop. Each senses something. Each triggers something. None of them is a one-time fix.
Feedback loops have specific requirements. They need a sensor (the source of the signal). They need a threshold (when the signal becomes actionable). They need a response (what happens when the threshold is crossed). They need an owner (who confirms the response happened). Without all four, the loop is incomplete and will not prevent.
The prevention section also names the assumptions the loops depend on. Continuous evaluation assumes the suite is current. Detectors assume the right baselines. Pact reviews assume the right reviewers. Each assumption has its own decay condition. The postmortem names the decay conditions so future operators can see when an assumption has gone stale.
A test for whether the prevention is real: in a year, when the team has rotated and no one in the room remembers this incident, will the feedback loops still fire? If yes, the prevention is structural. If no, the prevention depends on memory and will fail when memory does. Structural prevention transcends teams. Memorial prevention does not.
The prevention section is also where the postmortem connects to the agent's ongoing trust signal. A well-prevented incident leaves the agent's record stronger over time. The composite score recovers as clean operational history accumulates. The pact compliance record annotates the resolution. Future counterparties querying the trust oracle see the incident and the response together, and the response is the evidence that the operator takes accountability seriously. This is how transparency becomes a trust-building asset rather than a reputational liability.
The Blameless Discipline
Blamelessness is not a euphemism. It is a specific operational discipline that produces better postmortems and better systems. The discipline has three rules.
Rule one: name conditions, not people. When a contributing factor involves a human decision, describe the conditions under which the decision was made, not the human who made it. The condition is what the team can change. The human is what gets defensive. A postmortem that consistently describes conditions reads as analytical and produces structural fixes. A postmortem that names humans reads as accusatory and produces interpersonal damage.
Rule two: assume good intent. Every person involved in the incident, from the engineer who made the change to the on-call who handled the response, is assumed to have made the best decision available with the information they had at the time. Hindsight is not allowed. If a decision looks obviously wrong in retrospect, the postmortem must reconstruct what the decision-maker knew at the time before judging.
Rule three: focus on the system. The system includes the code, the pact, the tools, the evaluation suite, the deploy pipeline, the team structure, the training, and the documentation. Every contributing factor is a property of the system. Every fix is a change to the system. The system is what the team has authority over. Individuals are what the team has politeness toward.
Blamelessness is sometimes mistaken for excuse-making. It is the opposite. Blameless postmortems hold the system accountable in a way that named-person postmortems do not. The system cannot blame back. The system cannot quit. The system can only be improved or not improved, and the postmortem is the artifact that decides which.
Teams that adopt the blameless discipline see two effects in the first quarter. The number of postmortems goes up because people stop hiding incidents. The depth of postmortems goes up because participants are more willing to surface uncomfortable truths. Both effects are good. The aggregate is a team that learns faster and makes fewer repeat mistakes.
Reviewing And Publishing The Postmortem
The postmortem is not done when the author finishes the draft. It is done when it has been reviewed and published. Review and publication are quality gates that prevent the author's blind spots from becoming the team's blind spots.
Review has three readers. The agent owner, who validates the technical claims and the fixes. A peer from outside the immediate team, who tests the postmortem against the test of usefulness for someone unfamiliar with the agent. The incident commander, if different from the author, who validates the timeline and the response account.
Each reviewer reads with a specific lens. The agent owner asks: do the root causes and fixes match what I know about the agent. The peer asks: would I learn something useful from this if I were running a different agent. The incident commander asks: is the timeline accurate to what actually happened, and are the response choices represented fairly. Each lens catches a different class of error. The composite catches most.
Publication has a defined audience. Internal stakeholders who need the operational learning. External counterparties who were impacted, in summary form. The platform, if the platform's pact infrastructure is referenced. The dispute mechanism, if a dispute is in play. Publication is not optional. The postmortem that is reviewed and not published is institutional knowledge held by three people; the postmortem that is published is institutional knowledge held by the institution.
The published postmortem becomes part of the agent's permanent record in the trust layer. It is queryable by future counterparties. The combination of the incident, the postmortem, and the verified fix is structured evidence of how the operator handles failure. This evidence is competitive. Operators with publicly visible postmortem hygiene will increasingly attract counterparties who want to know what happens when things go wrong, not just what happens when things go right.
Artifact: The Agent Postmortem Template
The artifact for this post is the five-section postmortem template. Fill it out for every agent incident.
Header: Incident identifier, agent identifier, pact version, date and time of detection, date and time of resolution, incident commander, postmortem author, reviewers.
Section 1 — Timeline: Chronological entries with timestamp, description, source. Include non-events. Distinguish evidentiary entries from reconstructed ones. Tight; one to two pages.
Section 2 — Root cause in three layers: Proximate cause with the specific sequence. Contributing causes (pact under-specification, tool over-permission, evaluation under-coverage) each with evidence. Systemic cause naming the operations practice property that allowed the contributing causes to coexist.
Section 3 — Contributing factors: Human and process conditions named as system properties, not as individual failures. Review practices, documentation gaps, time pressures, staffing, tooling limitations.
Section 4 — Fix in three layers: Proximate fix, contributing fix, systemic fix. Each with owner, target date, verification step. Verifications include replay against incident snapshot.
Section 5 — Prevention through closed feedback loops: Each loop with sensor, threshold, response, owner. Include the assumptions the loops depend on and the conditions under which the assumptions decay.
Footer: Wins (what worked well during detection, containment, response). Items not being fixed and why. Permanent record entry for the trust layer.
The template is filled out within seven days of incident closure. Reviewed by agent owner, peer, and incident commander. Published to defined audience. Archived in the agent's permanent record.
Counter-Argument: The Template Is Too Heavy For Small Incidents
The most common objection to a five-section template is that it is overkill for small incidents that get resolved in an hour. The objection deserves a nuanced response.
For genuine micro-incidents, where no counterparty was impacted, no slashing occurred, and the fix was a single-line change, a lighter version of the template is appropriate. Three sections instead of five: timeline, root cause and fix combined, prevention. The same blameless discipline applies. The template scales down without losing its structure.
The objection becomes weak when applied to incidents that touch counterparties or trigger disputes. For these, the full template is the minimum. The audience for the postmortem includes parties who were not in the room, and the structure is what makes the postmortem comprehensible to them. Skipping sections for an external-facing incident is what produces postmortems that read like internal memos and fail their external audience.
The deeper objection is that postmortems take time and the team has a backlog of work. The defense is that postmortems are work. They are the work of preventing the next incident, which is the highest-leverage work the team can do. Teams that treat postmortems as overhead accumulate incidents until the cumulative cost is undeniable, at which point the team is in crisis mode and cannot run good postmortems anyway. Run them when you can. Build the discipline. The compounding pays.
The last version of the objection is that the template is biased toward technical incidents and underrepresents non-technical ones. For agent incidents this is rarely true; the technical and the operational are tightly coupled. For incidents that are primarily about communication, vendor, or process failures, adapt the template by reframing the contributing causes and contributing factors sections. The structure transfers. The vocabulary adjusts.
What Armalo Does
Armalo provides the substrate that makes detailed postmortems feasible. The audit trail captures every mutating action with structured fields, agent identifier, pact version, and timestamp, so the timeline section is reconstructed from authoritative data rather than from imperfect memory. The composite score history shows the trust signal across the incident, with the twelve-dimension breakdown that supports root cause hypothesis. The dispute mechanism produces structured filings and resolutions that integrate into the timeline. The bond and slashing records, on-chain in USDC on Base L2, are immutable evidence for the financial fields. The Trust Oracle is where the published postmortem's permanent record lives, queryable by future counterparties. The platform does not write your postmortem. It does ensure the inputs your postmortem needs are available, structured, and trustworthy. The discipline of postmortem hygiene is yours; the data substrate is ready.
FAQ
Should every incident produce a postmortem? Every incident that touched a counterparty, triggered a dispute, slashed a bond, or produced a five-or-more-point composite score drop. Smaller incidents produce a lighter version.
Who owns the template? A defined function in the operations team. Without an owner, the template drifts. The owner reviews the template quarterly and updates it based on what has been learned.
How long should the full template take to fill out? Two to four hours for the author. One hour each for the reviewers. Total elapsed time is one to two weeks because of review cycles. The discipline is to keep the elapsed time bounded.
What if the team disagrees on the root cause? Document the disagreement. Multiple plausible root causes are named. Fixes address the most likely one and a contingency for the alternatives. The postmortem becomes more honest, not less.
How do we prevent postmortems from becoming performative? The artifact requirement. Each section requires evidence. Each fix requires verification. Each prevention requires a closed loop. Performance cannot survive evidence.
Should postmortems be public? Internal always. External in summary form when counterparties were impacted. Fully public for high-stakes operators who want to use transparency as a trust signal. The trust layer makes structured summaries available regardless.
Bottom Line
Five sections, blameless framing, structured evidence. Timeline. Root cause in three layers. Contributing factors as system properties. Fix in three layers. Prevention through closed feedback loops. Run the template for every agent incident. Publish the result to the right audience. Archive it in the agent's permanent record. The cumulative library becomes the institutional memory that turns every incident into durable improvement. The teams that practice this discipline have fewer repeat incidents, faster recovery, and stronger trust signals over time. The teams that do not have postmortems that produce nothing and incidents that produce stories.
Continue exploring
Explore more articlesComments
Loading comments…