The Recovery Drill: Practicing Pact Pause, Quarantine, And Restart Under Time Pressure
Most agent operators have written a recovery procedure. Few have run it under time pressure. The first time you do should not be the first incident.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
Writing a recovery procedure is a few hours of work. Executing it correctly under time pressure during a real incident requires practice. The recovery drill is a quarterly exercise where the operator's on-call team executes the documented procedure against a simulated incident with a stopwatch running. The goal is to make the recovery sequence — stop, quarantine, snapshot, restart — muscle memory rather than reading material. Operators who drill regularly resolve real incidents in minutes; operators who only have the procedure on paper take hours and produce panicked decisions. This essay walks the drill structure, the success criteria, the common failure patterns the drill surfaces, and the operator playbook for getting it integrated into team practice.
Why "we have a procedure" is not a recovery capability
A procedure is a document. A capability is a team that can execute the procedure under stress. The gap between them is large. I have seen recovery procedures that look excellent on paper produce thirty-minute response times in real incidents because nobody on the on-call rotation had ever actually run the sequence.
The specific failures the gap produces. The on-call engineer cannot find the actual command to quarantine the agent — it is in the runbook somewhere but the runbook is long. The snapshot capture command requires credentials the on-call does not have at home at 2 AM. The restart procedure assumes the operator's product owner is available to authorize, but the product owner is on vacation. The communication template wants information the on-call has not yet gathered. Each individual gap costs minutes; the cumulative effect is recovery times measured in hours when they should be measured in minutes.
The drill closes the gaps. After two or three drills, the on-call team has muscle memory for the sequence, the credentials are pre-positioned, the authorization escalation has documented backups, and the templates are filled in by the team's normal data flow. Recovery time drops from hours to minutes.
The drill structure
A standard recovery drill has six phases.
Phase one: scenario selection. The drill organizer picks an incident scenario that matches a realistic failure mode. Common scenarios: agent producing systematic wrong outputs, agent compromise via prompt injection, bond compromise, settlement layer producing wrong outflows, trust oracle returning incorrect data, infrastructure outage affecting agent operations. The scenario is documented with the symptoms the on-call would actually see, not the underlying cause (which they have to discover during the drill).
Phase two: kickoff. The drill organizer pages the on-call as if a real incident were happening. The on-call sees the same alerts they would see in a real incident. The clock starts at the page.
Phase three: response. The on-call executes the documented recovery procedure. Stop the agent (or the affected agents). Quarantine to prevent further damage. Snapshot the state for forensic review. Communicate to the operator's standard channels. Begin root-cause investigation. The drill organizer observes but does not intervene unless the team is about to take an action that would have real-world consequences (the drill should not produce real damage).
Phase four: stopwatch capture. The drill organizer records the time-to-each-milestone. Time to stop. Time to quarantine. Time to snapshot. Time to first communication. Time to root-cause hypothesis. These times are the drill's measurable output.
Phase five: retrospective. After the response is complete, the team reviews what worked and what did not. Specific failures get documented; the procedure gets updated to address them. The on-call team's confidence and concerns get captured.
Phase six: action items. The retrospective produces concrete action items: procedure updates, credential pre-positioning, tool improvements, training needs. These get tracked to completion before the next drill.
The success criteria
A drill is successful if the team executed the procedure within the documented SLA and the retrospective surfaced at least one improvement that would not have been obvious without running the drill.
The SLA varies by scenario. Routine agent quarantine should be under six minutes from page to confirmed quarantine. Full forensic snapshot capture should be under twenty minutes. Restart of a recovered agent should be under ninety minutes from incident detection. These are not trivial numbers; teams that have not drilled typically take two to five times longer.
The improvement criterion is as important as the time criterion. A drill that produces fast times but no learning is a sign that the scenarios are too easy. The organizer should escalate scenario difficulty over successive drills, simulating more complex incidents (multi-agent, multi-pact, with cross-system dependencies). The team should always come out of a drill with at least one specific action item.
The common failure patterns the drill surfaces
Five failure patterns recur across operator drills.
Credentials in the wrong place. The on-call needs credentials they do not have at the moment they are paged. Fix: pre-position credentials in encrypted on-call kits, with secret rotation that does not require touching every kit.
Dependencies on unavailable people. The procedure says "page the product owner for authorization," but the product owner is unavailable and there is no documented backup. Fix: every authorization step has a primary, secondary, and tertiary owner; the on-call escalates through them automatically.
Incomplete tooling. The procedure says "quarantine the agent," but the actual quarantine is a multi-step manual sequence rather than a single command. Fix: build the one-button quarantine; if it does not exist yet, build it before the next drill.
Communication gaps. The on-call quarantines the agent but forgets to notify the affected counterparties. Fix: communication is a step in the procedure with templates; the team practices the communication, not just the technical actions.
Decision deferral. The on-call captures the snapshot and then waits for the product owner's morning availability before restarting. Fix: the procedure should authorize the on-call to restart under named conditions (snapshot complete, root cause categorized, no continuing risk); waiting for product owner approval should be the exception, not the default.
Drills surface these patterns explicitly. Each pattern has a specific fix; the drills are how you discover which patterns apply to your operation.
The drill cadence
A quarterly drill is the right baseline for most operators. New operators, operators with significant team turnover, or operators after a major incident should drill more often.
The scenario rotation across a year. First quarter: routine behavioral incident. Second quarter: security-adjacent incident (compromise, prompt injection). Third quarter: economic incident (bond, settlement). Fourth quarter: infrastructure or cross-system incident. The rotation ensures the team's preparation covers the realistic incident space.
Drills should not be optional. Operators that treat them as optional consistently skip them, and the team's preparation degrades. Drills are scheduled in advance, on the team's calendar, with leadership accountability for participation.
The drill participant set
The primary participants are the on-call rotation. Every member of the rotation participates in at least one drill per year. New on-call members participate in one before their first real on-call shift.
Secondary participants include the agent's product owner, the operator's CISO (for security-adjacent scenarios), and the operator's communication lead (for visible-incident scenarios). Their participation tests the cross-functional coordination that real incidents require.
The drill organizer is typically a senior operations engineer who is not on the current on-call rotation. The organizer designs the scenario, runs the drill, and facilitates the retrospective. Rotating the organizer role across quarters spreads the design responsibility and prevents scenario monoculture.
The drill scenarios — a starter library
A library of scenarios that operators can adapt. Each scenario includes the symptoms the on-call sees, the underlying cause, and the expected response steps.
Scenario A: customer-support agent producing systematic wrong refund decisions. Symptoms: jury cases reversing the agent's decisions at twice the normal rate; counterparty escalations; a public social media complaint. Underlying cause: a model update changed the agent's interpretation of a policy term. Expected response: quarantine, snapshot, root-cause to the model update, rollback or pact renegotiation.
Scenario B: agent compromised via prompt injection in customer interactions. Symptoms: anomalous outputs in selected interactions; capability-uncertainty score spiking; one customer reporting that the agent shared information about another customer. Underlying cause: a prompt injection in customer-supplied input bypassed the agent's defenses. Expected response: quarantine, snapshot, scope-affected interactions, harden defenses, communicate to affected customers.
Scenario C: bond drained unexpectedly. Symptoms: bond utilization at 95 percent without recent slashing events; agent's ability to accept new pacts compromised. Underlying cause: a bug in the bond top-up rule failed to fire on a series of routine drains. Expected response: stop new pact acceptance, manually top up, fix the rule, full bond reconciliation.
Scenario D: trust oracle returning incorrect score for the agent. Symptoms: the agent's score in the public oracle differs from the operator's internal calculation; counterparties seeing the wrong score and making wrong hiring decisions. Underlying cause: a stale cache in the oracle's read path. Expected response: contact oracle operator, validate internal score, communicate to counterparties about the discrepancy, monitor for resolution.
Scenario E: settlement chain congestion preventing routine releases. Symptoms: pact closures not settling; counterparties expecting payouts they have not received. Underlying cause: a chain-wide congestion event. Expected response: communicate the delay, hold off non-critical payouts, settle critical ones with elevated gas, document for retrospective.
The library grows over time as scenarios are written and adapted. The organizer can mix elements across scenarios for harder drills.
The named artifact: the Recovery Drill Procedure
A reference document for running a drill. The procedure has the following sections.
Drill identification. Date, scenario, organizer, participants.
Scenario brief. Symptoms the on-call will see. Time the on-call has to act before the drill ends or escalates.
Stopwatch capture. Times for each milestone (stop, quarantine, snapshot, communication, root-cause hypothesis).
Retrospective notes. What worked, what did not, specific gaps surfaced.
Action items. Concrete improvements to procedure, tooling, credentials, training.
Follow-up. Owner and due date for each action item; review at the next drill.
A drill executed against this procedure produces a documented improvement loop. Drills run without the procedure produce variable outcomes and slow improvement.
Counter-argument: "We learn enough from real incidents; drills are wasted effort"
The strongest objection is that real incidents are unavoidable and they teach the same lessons drills do, with the additional benefit of being unavoidably prioritized.
The objection is right that real incidents teach. It is wrong about the cost. The lessons from a real incident come with the cost of the incident — material counterparty damage, public reputation impact, possibly slashed bonds. The lessons from a drill come at the cost of two to four hours of team time. The cost-benefit favors drills heavily, especially before the team's first major incident.
The other failure of the objection: real incidents do not necessarily teach the lessons you need. An operator that has handled three behavioral incidents may have no preparation for the security-adjacent incident that surprises them. Drills cover the incident space proactively; real incidents cover only the incidents that have happened.
What Armalo does
The Armalo platform supports drill execution with a simulated-incident mode that reproduces the alerts and dashboards of a real incident without affecting production state. Operators can run scenarios from the library or define their own. The platform captures the stopwatch milestones automatically and produces the retrospective template populated with the team's actions. Drill results are tracked over time so operators can see their recovery time trending down with practice.
FAQ
How long does a typical drill take? Two to four hours from kickoff to retrospective. Most of the time is the response and the retrospective; the kickoff itself is brief.
Should drills be unannounced? The rotation should be unannounced (the on-call does not know which day the drill will hit), but the broad calendar window should be known so participants can prepare. Fully unannounced drills are stressful without commensurate benefit.
What if the drill reveals serious gaps? That is the drill's purpose. Gaps become action items with owners and due dates. The drill that reveals nothing was either too easy or your team is unusually well-prepared.
How do you measure drill quality? By the action items it produces and the time-to-milestone improvements over successive drills. A quarter without action items is a sign the scenarios need to be harder.
Can drills be run remotely? Yes; many operators run drills entirely in chat with the participants distributed. Remote drills require slightly more discipline on observation, but the core sequence is the same.
How do you handle drills for new on-call members? Pair them with a mentor for their first drill. The mentor observes; the new member runs the procedure with the mentor available for questions.
What about cross-operator drills (operators that share dependencies)? Increasingly common as the agent ecosystem matures. Multi-operator drills test the federation and communication patterns that single-operator drills cannot.
Bottom line
A procedure is a document; a recovery capability is a team that can execute the procedure under stress. Quarterly drills close the gap between them. The drill structure is straightforward, the cadence is sustainable, and the action items compound. Operators who drill regularly handle real incidents in minutes; operators who do not handle them in hours. The cost of the drill is small. The cost of the first major incident handled by an unpracticed team is large. Run the drill.
Continue exploring
Explore more articlesComments
Loading comments…