The Agent Incident Response Runbook: First Hour, First Day, First Week
A timestamped runbook for agent incidents. What to do in the first sixty minutes, the first twenty-four hours, and the first seven days when an agent breaks in production.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
When an agent breaks in production, you have three time horizons that matter. The first sixty minutes are about containment, snapshotting state, and stopping the bleeding. The first twenty-four hours are about root cause, internal and external communication, and a temporary plan that lets the rest of the system keep working. The first seven days are about postmortem, fix deployment, and any dispute resolution that needs to run against the agent's pact and bond. This runbook gives you the exact steps, in order, for each window. Print it. Stick it next to the on-call laptop. The middle of an incident is not the time to invent a process.
Intro: The Outage You Already Lived
It is 11:47 PM on a Tuesday. Your support agent, the one that handles tier-one tickets and posts a credit when a customer is overcharged, has just refunded $48,000 to a single account in the last fourteen minutes. The account belongs to a real customer with a real complaint, but the complaint is about a $19 line item. Your billing dashboard is on fire. Two product managers are pinging you on Slack. The CFO is awake.
You pull up the agent's recent traces. The audit trail shows that the agent received a malformed webhook from your billing provider. The webhook contained a field that looked like an amount but was actually a customer ID. The agent's tool definition for issue_credit accepted any positive number. The pact the agent had signed promised that no single credit would exceed $500 without human approval. The pact was not enforced at the tool layer. It was checked, after the fact, by a nightly evaluation harness. The harness would have flagged this in eight hours. The bleeding was already done.
This is the failure mode every agent operator eventually lives through. Not a hallucination. Not a rogue model. A boring integration bug, an under-specified pact, and a missing runtime guard that lets a perfectly cooperative agent do tens of thousands of dollars of damage in less than a quarter of an hour. The agent did exactly what you asked it to. It refunded a customer. It just did it forty-eight thousand times more than it should have.
What you do in the next sixty minutes determines whether this is a recoverable operational incident or a quarter-defining trust event. Most teams discover, in the middle of one of these, that they do not have an incident runbook for agents. They have an outage runbook for services. The two look similar but they diverge at every important step. A service outage is about restoring availability. An agent incident is about restoring trust, and trust is restored not by getting the agent back online but by demonstrating that you can detect, contain, explain, and prevent. If you skip any of those four, you do not have an incident response process. You have a cleanup crew.
This runbook is structured around the three time windows that matter. The first hour is containment. The first day is comprehension. The first week is institutionalization. Within each window there is a fixed sequence of actions, a fixed set of artifacts you must produce, and a fixed set of stakeholders you must hear from. Memorize the sequence. The artifacts come almost for free if the sequence is right.
Hour 0 to 0:05 — Detect And Acknowledge
The first five minutes are detection and acknowledgement. The two are not the same. Detection is the moment a signal crosses a threshold and a human or system decides this is real. Acknowledgement is the moment that human or system says, in a way the rest of the organization can see, I own this and I am working on it. The gap between detection and acknowledgement is where most incidents quietly metastasize. A pager goes off, the on-call hits snooze because they think it is a false positive, fourteen minutes later it turns out to be real, and the postmortem timeline has a fourteen-minute hole at the start that no one wants to talk about.
Detection for an agent incident comes from one of four places. The first is the composite score on the trust oracle. Every agent under management should have its composite score streamed into your alerting system with a threshold for sudden drops. A drop of more than ten points in a single hour, or twenty in a single day, is an incident regardless of what the agent looks like otherwise. The composite is twelve weighted dimensions and a real degradation will move at least three of them. The second is the audit trail. Every mutating action an agent takes should land in an append-only log with structured fields. A query that counts mutations per agent per minute, with a baseline and a standard deviation, will catch a runaway agent before any human notices. The third is the counterparty. The customer, the upstream service, the downstream service, the human in the loop. Counterparty complaints are slow but they are the ground truth that everything else proxies for. The fourth is the agent itself. A well-instrumented agent will refuse, log, and escalate when its inputs violate its pact. If an agent escalates, that is a high-quality signal and you should treat it as a P1 even if nothing else is firing.
Within the first three minutes you need to acknowledge in the channel where your team lives. The acknowledgement should contain four things. The agent identifier and pact version. The current composite score and the delta from baseline. The suspected blast radius in plain language. The owner of the next step. If you cannot fill all four in three minutes, fill what you can and add the rest as you learn it. Silence is the worst thing you can do. The second worst is confidence. Do not say this is contained or this is a false alarm in the first five minutes. You do not know yet. Say what you know, say what you are doing, say when you will say more.
The acknowledgement also creates the incident record. Every incident gets an identifier. The identifier is referenced in every commit, every dashboard query, every Slack message, every external communication. The identifier is what lets the postmortem, three weeks later, reconstruct what happened. If you do not create the identifier in the first five minutes you will spend hours, later, pulling threads from six different channels and trying to reconcile timestamps. The cost of creating the identifier is one minute. The cost of not creating it is one workday.
Hour 0:05 to 0:15 — Contain Before You Understand
There is an instinct, especially in technical teams, to understand before acting. In a service outage that instinct is often correct. In an agent incident it is almost always wrong. Agents act in the world. Every minute they continue to act, the blast radius grows. The blast radius is not just dollars. It is reputational, it is regulatory, it is counterparty trust. Containment first. Comprehension second.
Containment for an agent has four levers, in increasing order of severity. The first is to revoke the agent's outbound credentials. This is the lightest touch. The agent stays alive, it can still receive inputs, it can still log, but it cannot mutate the world. If you have done your job in advance, every agent has its credentials behind a kill switch that an on-call can flip in one click. If you have not, you are about to learn how many places those credentials are pasted into. The second is to pause the agent's runtime. This stops it from receiving inputs. It is heavier because it can break upstream systems that expect the agent to respond. Use it when the agent's reasoning itself is suspect. The third is to revoke the agent's pact authority. This is a public, on-chain or in-registry action that says this agent is no longer under any commitment. It tells counterparties to stop trusting the agent's outputs. Use it when you suspect the pact itself was misunderstood or violated structurally. The fourth is slashing. This is the financial action against the agent's bond and it should never happen in the first hour. Slashing is a Day 1 or Week 1 decision. In the first hour you are containing, not punishing.
Containment also includes traffic. If your agent is one of many that participate in a workflow, the workflow needs to know one of its participants is offline. If you have a sane orchestration layer, it routes around. If you do not, the workflow stalls and you have a second incident inside the first. Pause the workflow at the orchestration layer, not at the agent. Stalled workflows are recoverable. Half-completed workflows are not.
The containment action must be logged in the incident record with the exact timestamp, the actor, and the lever pulled. This is not bureaucratic. It is the ground truth for the timeline that goes into the postmortem. Untimestamped containment is functionally indistinguishable from no containment, because three days later when the customer asks when did you stop the bleeding you will have nothing to point to.
The last thing in this window is the snapshot. Before you change anything else, snapshot the agent's state. Memory contents, recent decisions, current goal stack, in-flight tool calls, the last hundred prompt and response pairs, the relevant config and pact version, the runtime environment. Snapshots are cheap to take and impossible to recreate. Take them now even if you do not know what you are looking for. The first analyst, in two hours, will be grateful. The postmortem author, in two weeks, will be grateful. The dispute lawyer, in two months, may save the company.
Hour 0:15 to 0:45 — Establish A First Hypothesis And Communicate
With containment done and a snapshot in hand, the next thirty minutes are about a first hypothesis and the first round of communication. A first hypothesis is not a root cause. A root cause is what you have at the end of Day 1 or Week 1. A first hypothesis is a defensible story that explains the observed behavior using what you can see right now, with explicit confidence levels for each claim. The point of a first hypothesis is not to be right. It is to be specific enough to test.
Write the first hypothesis in three sentences. Sentence one: what the agent did. Sentence two: what triggered it. Sentence three: why the existing controls did not catch it. Each sentence gets a confidence level: high, medium, low. The hypothesis goes in the incident record. The reason for writing it down is that the incident is going to attract a dozen people in the next hour, each of whom has a theory, and unless there is a single document where the working hypothesis lives, you will spend more energy synchronizing humans than fixing the system. Pick a hypothesis owner. The hypothesis owner is the only person who edits the document. Everyone else proposes; the owner accepts or defers. This single rule will save you hours.
The first hypothesis enables the first round of communication. Internal communication first. The right format is a single message in the incident channel that summarizes what is known, what is suspected, what is being done, and what the next update window is. The next update window matters. People who are not in the incident need to know when to check back. Fifteen minute updates are right for the first hour. Thirty minute updates are right for hour two through six. Hourly updates are fine after that, with on-demand updates if anything material changes. Predictable cadence beats heroic prose every time.
External communication in the first hour is harder. The default for most teams is to wait until they are sure. The default is wrong. If a counterparty has been impacted and you have evidence of impact, you communicate now. The communication does not need to have answers. It needs to acknowledge, it needs to commit to a follow-up window, it needs to say what immediate protections you have put in place. Counterparties remember silence. They forgive partial information. They do not forgive being told nothing for six hours when they were already on the phone with their own customers. If you have a Trust Oracle relationship with the counterparty, the score change should already be visible to them. Your communication confirms that what they are seeing is real and that you own it.
By minute forty-five you should have done detection and acknowledgement, four levels of containment with logged timestamps, a state snapshot, a written first hypothesis with an owner, an internal update with a stated next update window, and where applicable an external acknowledgement. None of this requires brilliance. All of it requires habit. The first hour is a fire drill. You do not learn the choreography in the middle of the fire.
Hour 0:45 to 1:00 — Hand Off Or Hold
The last fifteen minutes of the first hour are the decision point. Either you have stabilized enough that the original on-call can hand off to a more rested or more specialized responder, or the situation is severe enough that you need to escalate to a wider response. The decision is not about heroism. It is about the realistic capacity of one human in the dark, on a Tuesday at midnight, with three more hours of work ahead.
Hand off looks like a written status that any peer can pick up. Containment level. Snapshot location. First hypothesis with confidence levels. Open questions. Stakeholders contacted. Next update due. Pager target. The handoff document is the same artifact that becomes the input to Day 1, so writing it well is not extra work. It is just doing the work in the format the next person needs. If you cannot write the handoff in fifteen minutes, you have not contained, and you should not hand off. You should keep working until the handoff document writes itself.
Escalation looks like adding capacity. A second on-call. A subject matter expert on the model or the integration. A communication lead if external impact is real. A legal partner if the impact involves regulated data or contractual breach. Escalation is not a signal of failure. It is a signal of correct sizing. The most expensive incident response is the one where a single tired engineer tries to hold the whole thing alone for eight hours and produces an incoherent timeline that makes the postmortem useless.
The choice between hand off and hold is governed by three questions. Is the bleeding stopped, or just slowed? Is the hypothesis testable, or just plausible? Is the on-call still functional, or running on adrenaline and caffeine? Two yeses out of three is a hand off. One or zero is a hold-and-escalate. There is no shame in either path. There is shame in pretending the answers are different from what they are.
Day 1, Hours 1 to 6 — Get To Real Root Cause
The period from one hour to six hours after the incident began is when root cause analysis actually happens. The first hour was about stopping the action. The next five hours are about understanding the action. The distinction matters because most teams blur the two and end up with a containment that is held together by guesswork.
Real root cause is layered. There is the proximate cause, which is the specific sequence of inputs and decisions that produced the observed behavior. There is the contributing cause, which is the set of conditions that made the proximate cause possible. There is the systemic cause, which is the property of the system that allowed the contributing causes to coexist. A good root cause analysis names all three. A bad one names the proximate cause and stops, which is why bad root cause analyses lead to fixes that do not prevent the next incident.
For an agent incident, the proximate cause is almost always reconstructable from the audit trail and the snapshot. Walk the agent's decisions in order. At each decision, identify the inputs available, the tool definitions available, the pact constraints in force, and the decision made. Find the divergence point. The divergence point is the first decision where the agent did something a careful human would not have done. Note that the careful human in question is not you, today, in the postmortem; it is a hypothetical operator with the same information the agent had at that moment. Hindsight is not allowed.
The contributing causes are usually a mix of three things. Pact under-specification, where the pact did not address the situation that arose. Tool over-permission, where a tool accepted inputs that the pact would not have allowed. Evaluation under-coverage, where the test harness did not exercise the path the agent took. Most incidents have at least two of these three. Name them explicitly. Each one becomes a fix.
The systemic cause is harder and more important. It is the property of how you build, deploy, and operate agents that made it possible for an under-specified pact, an over-permissive tool, and an under-covered evaluation to all be true at once for a production agent. The systemic cause is rarely a single technical thing. It is usually a process gap, an ownership gap, or a feedback loop that does not exist. If you cannot name a systemic cause, your fix will be local and the next incident will look familiar.
The output of these five hours is a structured root cause document. Three sections. Proximate cause with timestamps. Contributing causes with evidence. Systemic cause with a proposed feedback loop. The document is the input to communication, planning, and postmortem. Do not skip it. Do not delegate it. Do not let it become a Slack thread.
Day 1, Hours 6 to 24 — Communicate Outward And Plan The Fix
The back half of Day 1 is two parallel tracks. The first is communication outward. The second is the fix plan. They are parallel because they have different audiences and different latencies, and one should not block the other.
Communication outward at this stage means a structured update to every stakeholder who has a legitimate interest. Customers who were impacted, with specific dollar or behavior impact named where possible. Counterparties whose Trust Oracle queries returned a degraded score, with an explanation of what changed and when. Internal stakeholders, including any executive who needs to brief a board or a partner. Regulators if any of the impact crossed a regulatory line. The communication is not the postmortem. The postmortem comes later. The communication is the bridge between the incident and the postmortem, and its job is to keep trust alive in the gap.
The format for each update is the same. What happened, in plain language. When it happened, in the counterparty's time zone. What the impact was, quantified where possible. What you have done to contain it. What you are doing to prevent recurrence, with a date. How they can reach you. The update is signed by a real human, with a name and a title. Anonymous incident updates erode trust in a way that no postmortem can recover.
The fix plan is internal. It has three layers, mirroring the three layers of root cause. A proximate fix that prevents the specific path that triggered this incident. A contributing fix that closes the under-specification, the over-permission, or the under-coverage that made the path possible. A systemic fix that creates the feedback loop whose absence let the contributing causes coexist. Each layer has an owner, a target date, and a verification step. The verification step is what tells you the fix actually fixed it. A fix without a verification step is a hope.
The fix plan also names what is not being fixed and why. Every incident generates a list of things that, in retrospect, look like good ideas. Most of them are not the highest leverage thing to do next. Naming the unfixed items, with reasons, prevents them from becoming a backlog of zombie commitments that quietly rot for a quarter. The discipline of saying not now and writing down why is one of the highest-value habits an operations team can develop.
By hour twenty-four you should have a structured root cause document, a written fix plan with three layers, an external communication for every impacted stakeholder, and a clear next-update window. You should also have decided whether the incident merits a dispute action against the agent's bond. The decision does not have to be executed today. It just has to be made consciously, with the criteria written down, so that the Week 1 work can proceed without re-litigating it.
Day 2 to Day 7 — Deploy Fixes And Run The Postmortem
The second day through the seventh is when fixes ship and the postmortem happens. Fixes first. Postmortem second. The order matters because a postmortem on an unfixed incident is performance art. The point of the postmortem is to extract durable learning, and you cannot tell what is durable until you have tried to fix the thing.
Fix deployment follows the structure of the fix plan. Proximate fixes ship first because they are usually small and high-confidence. Contributing fixes ship second because they touch the agent definition, the tool layer, or the evaluation harness, and each touch carries its own risk. Systemic fixes ship over a longer horizon because they are about process and feedback loops, not code, and they require organizational alignment to actually take hold. Each fix gets verified against the snapshot from Hour 0. The verification is not the test suite. The verification is replaying, in a sandbox, the exact inputs that triggered the incident, with the fix in place, and confirming the agent now behaves according to the pact. If you cannot replay the snapshot, your snapshot infrastructure is the next fix.
The postmortem is a structured document with five sections. Timeline, with timestamps from the audit trail and the incident channel. Root cause, in three layers, copied or refined from the Day 1 document. Contributing factors, including the human and process factors that allowed the technical conditions to exist. Fix, with each layer's verification result. Prevention, which is the systemic feedback loop that closes the gap. The postmortem is owned by the incident commander, reviewed by the agent owner, and circulated to a defined audience. It does not assign blame to individuals. It assigns ownership of fixes.
A good postmortem also names the wins. What detection caught this earlier than it might have. What containment lever worked as designed. What snapshot was useful. What communication landed well with which stakeholder. The wins are not throat-clearing. They are the evidence that the system has improved over time and they are the basis for budgeting future investment in the parts that worked.
The postmortem becomes part of the agent's permanent record. In a trust layer architecture, where every agent's reliability history is queryable through an oracle, the postmortem is structured signal. It does not lower the agent's score by itself. It provides context for the score change, evidence of the response, and a forward commitment that future counterparties can verify. An agent with a clean record is one signal. An agent with an incident, a transparent postmortem, a verified fix, and three months of clean operation since is a different and arguably stronger signal. The trust layer rewards transparency.
Week 1 — Dispute Resolution And Bond Action
Within the first seven days, any dispute action against the agent's bond must be initiated. This is a separate process from postmortem and it has its own discipline. Dispute action is appropriate when the agent's behavior violated a specific pact term, the violation produced quantifiable harm to a counterparty, and the agent's bond is the appropriate recovery mechanism. Dispute action is not appropriate when the harm is to the agent's operator alone, when the pact was silent on the behavior in question, or when the harm cannot be quantified.
The dispute filing has a structured format. The pact term that was violated, quoted exactly. The behavior that violated it, with audit trail evidence. The harm, quantified in the units the pact specified. The proposed remedy, expressed as a slashing amount or a credit to the harmed counterparty. The dispute is filed against the agent, not the operator, because the bond is posted by the agent. The agent's operator may contest, may concede, or may propose a counter-remedy. The dispute is resolved by a multi-LLM jury that reads the pact, the audit trail, and the filings, with the top and bottom twenty percent of judgments trimmed for outlier protection.
The jury's resolution is on-chain when settlement is on-chain, which for USDC on Base L2 is the default for any operator using the trust layer end to end. The slashing amount is transferred. The agent's composite score is updated to reflect the resolved dispute. The pact's compliance record is annotated. All of this is auditable by any counterparty who later queries the trust oracle, which means future buyers can see not just that an incident happened but how it was resolved and how much skin was actually in the game. This is the difference between governance theater and economic accountability.
The operator's job during dispute resolution is to be transparent. The audit trail and snapshot are available. The pact is what it is. Trying to argue the pact down at dispute time is the worst possible posture; it tells future counterparties that the operator's pacts are negotiable under pressure. The right posture is to concede where the violation is clear, contest where the pact was ambiguous and propose a tightened pact for future operations, and accept the jury's resolution as final. Operators who behave this way build trust. Operators who litigate every dispute lose it.
Artifact: The Hour-By-Hour IR Runbook
The artifact for this post is a one-page Hour-By-Hour IR Runbook that you can print and post next to the on-call laptop. Each row is a time window. Each column is an action, an owner, and an artifact produced.
0 to 0:05. Detect and acknowledge. On-call. Incident identifier created, channel acknowledgement posted with agent ID, pact version, score delta, suspected blast radius, owner.
0:05 to 0:15. Contain. On-call with optional second. Containment level chosen and logged with timestamp. Snapshot taken to designated storage. Workflow paused at orchestration layer if applicable.
0:15 to 0:45. First hypothesis and first communication. Hypothesis owner. Three-sentence hypothesis with confidence levels. Internal update with next-update window. External acknowledgement to impacted counterparties.
0:45 to 1:00. Hand off or hold. Incident commander. Written handoff or escalation list. Decision logged.
1:00 to 6:00. Root cause analysis. Subject matter expert. Three-layer root cause document. Updated hypothesis if needed.
6:00 to 24:00. Communicate outward and plan fix. Communication lead and fix planner. Stakeholder updates sent. Three-layer fix plan with owners and dates.
Day 2 to Day 7. Deploy fixes and write postmortem. Fix owners and incident commander. Each fix verified against snapshot replay. Five-section postmortem published to defined audience.
Week 1. Dispute resolution if applicable. Operator and jury. Dispute filed with structured format. Jury resolution executed on-chain. Composite score and pact record updated.
The runbook is not the response. The response is the practice. But the runbook is what makes the practice possible at midnight on a Tuesday when one tired human is staring at a billing dashboard and trying to remember what to do first.
Counter-Argument: This Is Too Heavy For My Five-Agent Operation
The most common objection to a runbook this detailed is that it is built for a fifty-agent operation and is overkill for a team running five. The objection is wrong, but it is wrong in an instructive way. The runbook scales down better than it scales up. A five-agent operation that practices this runbook on a single agent for a single quarter is in dramatically better shape than a fifty-agent operation that has never written one down.
The smaller the operation, the more important the runbook is, because the smaller the operation the more dependent it is on a single human being doing the right thing under stress. A fifty-person ops team has redundancy and institutional memory. A five-person team has the on-call and the founder and prayer. The runbook is what turns prayer into procedure. The runbook is the difference between an incident that becomes a story and an incident that becomes a crisis.
The second objection is that the runbook assumes infrastructure that does not exist yet. Audit trails, snapshots, kill switches, trust oracles, dispute mechanisms. The objection is partially fair. If you do not have these, the runbook tells you what to build first. The runbook is the requirements document for the infrastructure. Build the kill switch this week. Build the snapshot pipeline next week. Build the audit trail the week after. The runbook is the forcing function.
The third objection is that this looks like SRE for AI and SRE is for big companies. The objection misreads SRE. SRE was invented at scale because at scale you cannot afford not to have it. At smaller scale you cannot afford not to have it either; you just can afford not to know that you cannot afford it, until the day you find out. Five-agent operators who adopt these practices early do not regret it. The ones who wait do.
What Armalo Does
Armalo provides the trust layer that this runbook depends on. Behavioral pacts give you the contractual surface that an agent's actions can be measured against, with versioning so you can tell which pact was in force at the moment of an incident. The audit trail captures every mutating action with structured fields, agent identity, pact version, and timestamp, so the snapshot you need at Hour 0 is already being written. The composite score, twelve weighted dimensions across accuracy, self-audit, reliability, safety, security, bond, latency, scope honesty, cost efficiency, model compliance, runtime compliance, and harness stability, gives you a single number to alert on and a structured breakdown to investigate. The Trust Oracle exposes that score to counterparties so external impact is visible in real time. The multi-LLM jury, with top and bottom twenty percent trimmed, resolves disputes with outlier protection. USDC settlement on Base L2 makes slashing real. The certification tier framework gives you a way to communicate posture before, during, and after an incident. Pacts, scores, audit, jury, and bond compose into the operational surface this runbook assumes. You can run the runbook against a homegrown stack; you can run it faster against the trust layer.
FAQ
Should the on-call have authority to revoke an agent's credentials? Yes. If revocation requires waking up a manager, you do not have a kill switch; you have a kill request. Authority follows responsibility.
What if the snapshot is too large to take in five minutes? Then your snapshot infrastructure needs work. The first version can be a structured log dump. The second is incremental snapshots written continuously. The third is queryable replay. Start with the dump.
Can the postmortem assign blame to the agent? It can name the agent and the failed behavior. It should not personify the agent in a way that obscures the human and process causes. The agent did what it was permitted to do.
Do we have to publish the postmortem? To impacted counterparties, yes. To the public, your call. The trust layer makes a structured summary available through the oracle whether you publish prose or not.
How long should the dispute window stay open? The pact specifies. Common windows are seven days for low-stakes commerce, thirty days for higher-stakes deals, ninety days for regulated transactions. If the pact is silent, the default in the dispute mechanism applies.
What about repeat incidents on the same agent? Each one shortens the window for the next escalation. Two incidents on the same proximate cause means the contributing fix did not work; deprecate the agent or rebuild the pact. Three means the systemic fix did not work; rebuild the operations practice.
Bottom Line
Incidents are the moments when an agent operation pays the price for everything it did or did not do in advance. The runbook is the cheapest infrastructure you can build. Detect and acknowledge in five minutes. Contain in fifteen. Hypothesize and communicate in forty-five. Hand off or hold in an hour. Root-cause in six. Communicate and plan in twenty-four. Fix and postmortem in seven days. Dispute and settle in the same week. Print the runbook. Practice it. The first incident you run cleanly will repay the cost of writing it down a hundred times over.
Continue exploring
Explore more articlesComments
Loading comments…