Severity Tiers For Agent Incidents: How To Decide Page-On-Call Versus Daily Digest
Most agent operations teams page on-call for everything or nothing. The right answer is a five-tier severity model with explicit triggers, escalation paths, and response SLAs.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
Most agent operations teams handle incidents at one of two extremes: page on-call for everything, which produces alert fatigue and delayed response to the incidents that actually matter, or page on-call for nothing, which produces serious incidents discovered only when a counterparty reports them. The right answer is a five-tier severity model — S1 through S5 — adapted from SRE practice for the specific failure modes of agent operations. This essay walks the model, the trigger criteria for each tier, the escalation path, and the response SLAs. Operators who implement it consistently respond to real incidents faster and waste less attention on noise.
Why agent incidents need their own severity model
The SRE severity model is well-established for traditional infrastructure incidents: production outages, data loss, security breaches, performance degradations. The same model applied directly to agent operations produces wrong calls. An agent that produces a confidently wrong answer to a high-stakes question has not produced a service outage in the SRE sense — the service is up, the latency is normal, the error rate is zero — but it has produced an incident that, by some measures, is worse than an outage. Outages are visible and recoverable; confidently wrong answers can propagate damage before anyone notices.
Agent incidents have their own failure surface and need their own severity model. The five tiers below are calibrated to the specific risks: behavioral failures (the agent did something wrong), trust failures (the agent's reputation surface is producing the wrong signal), economic failures (bonds, settlements, escrow), operational failures (the supporting infrastructure), and reputational failures (something visible is happening that affects the operator's standing).
S1: critical, page-on-call immediately
Triggers. Agent producing systematic, high-stakes wrong answers in production (not isolated incidents but a pattern). Bond compromised or stolen. Settlement layer producing wrong outflows. Trust oracle returning incorrect data about the agent. Public-visible incident with material reputational damage.
Response SLA. On-call paged within five minutes. Acknowledged response within fifteen minutes. Containment action initiated within thirty minutes. Executive notification within sixty minutes.
Containment patterns. Agent quarantine (cease all new pact acceptance, hold in-flight pacts pending review). Bond freeze (no slashing, no release until the incident is resolved). Settlement freeze (manual review for all outflows). Public communication if the incident is visible.
Examples. An agent that has been producing wrong refund decisions for the last forty-eight hours; a settlement layer transferring USDC to wrong recipient addresses; a trust oracle returning a fake high score for a low-quality agent.
S2: high, page-on-call within business hours
Triggers. Composite score drop greater than one hundred points without explained cause. Sustained anomalous behavior pattern (greater than one standard deviation from baseline) lasting hours. Dispute backlog growing faster than resolution. Bond utilization above ninety percent of capacity. Significant evaluator disagreement on recent jury cases.
Response SLA. On-call paged within thirty minutes during business hours, next business hour overnight. Acknowledged response within one hour. Investigation initiated within four hours. Daily standup mention until resolved.
Containment patterns. Reduced pact acceptance (cap new pacts at fifty percent of normal capacity). Increased evaluation cadence on the affected agent. Operator standby ready to escalate to S1 if the situation deteriorates.
Examples. An agent whose composite dropped from 880 to 760 over a week with no obvious cause; an unusual increase in customer escalations on an agent's outputs; a bond utilization spike that suggests bond exhaustion if the trend continues.
S3: medium, daily review
Triggers. Single-event evaluation failures. Composite score drift up to fifty points. Routine dispute filings within normal range. Bond utilization above seventy percent. Capability misclassification on individual cases.
Response SLA. Logged in the daily standup. Investigation by end of business day. Resolution or reclassification within forty-eight hours.
Containment patterns. None automatic; investigation may surface need for higher-tier action.
Examples. A single jury verdict that went against the agent in a case the operator believes was wrongly decided; one customer complaint about an agent interaction that was technically correct but tone-mismatched; a normal week of disputes resolving as expected with one slightly anomalous case.
S4: low, weekly review
Triggers. Capability fit questions raised by the operator's quarterly review. Pact terms approaching renewal with no clear renewal decision. Dispute outcome trends suggesting subtle calibration drift. Reputation dimension drift below the alarm threshold but trending unfavorably.
Response SLA. Mentioned in weekly review. Investigated by the end of the next sprint. Often produces a longer-term improvement initiative rather than an immediate fix.
Examples. The agent's response time has gradually crept up over a month, still inside SLA but trending toward the boundary; the agent's scope-honesty score has slowly declined while its accuracy has held; the next quarterly renewal of a pact is uncertain because the buyer has not engaged with renewal questions.
S5: informational, capture and aggregate
Triggers. Routine telemetry within normal bounds. Capability questions at the edge of scope that the agent handled correctly but that suggest scope expansion opportunities. Counterparty feedback that is positive but suggests refinements.
Response SLA. Captured in the operator's continuous improvement backlog. Reviewed in the monthly retrospective.
Examples. A counterparty mentioned in passing that they wished the agent could handle a related capability the operator has not formally added; the agent's latency has improved measurably over the last sprint without any specific work; a routine score increase from steady operation.
The escalation path
Incidents do not always start at the right tier. The escalation path moves them.
Upward escalation. An S3 that turns out to be the visible tip of an S1 should escalate as soon as the deeper picture emerges. Operators who hesitate to escalate produce delayed responses to real incidents. The bias should be to escalate when in doubt; the cost of an unnecessary S1 page is small, the cost of a missed escalation is large.
Downward de-escalation. An S2 that turns out on investigation to be a known false alarm pattern should de-escalate after the investigation. The de-escalation is logged with the false-alarm reason; if the same false alarm recurs, the operator's monitoring needs to be tuned.
Lateral movement. Some incidents move sideways across categories. A behavioral incident that turns out to be caused by a settlement layer issue moves from the agent's incident response to the settlement team's. The handoff is logged.
The response team composition by tier
Different tiers need different response teams.
S1 incidents pull the on-call operations engineer, the agent's product owner, the operator's CISO (for security-adjacent incidents), and the operator's legal counsel (for incidents with regulatory or contractual exposure). The team meets within an hour and stays in regular contact until containment.
S2 incidents pull the on-call operations engineer and the agent's product owner. Other roles join only if the investigation surfaces their relevance.
S3 incidents are handled by the on-call operations engineer with the agent's product owner consulted as needed.
S4 incidents are handled in the team's normal sprint cadence; no special response team.
S5 incidents flow into backlog grooming.
The communication discipline
Incidents need communication. The discipline varies by tier.
S1 incidents require executive notification within an hour, public communication if the incident is visible (typically within four hours of containment), counterparty notification for affected pacts, and a public postmortem published within ten business days of resolution.
S2 incidents require operator-internal notification within hours, counterparty notification for affected pacts, and a postmortem published in the operator's internal incident review within fifteen business days.
S3 incidents require logging in the standup and the operator's incident tracker; no external communication unless the incident has counterparty impact.
S4 and S5 incidents require logging in the appropriate review cadence; no external communication.
The named artifact: the Severity Tier Decision Table
A single-page reference that on-call uses to triage incidents in real time. The table has rows for each common incident pattern and columns for severity tier, immediate action, escalation criteria, and communication requirements.
Incident pattern. Examples: "composite score dropped 150 points," "agent produced wrong refund decision," "bond freeze hit," "counterparty disputes settlement."
Default severity tier. The starting tier; on-call may escalate based on context.
Immediate action. The first thing on-call does (often: quarantine, freeze, alert operator, page domain expert).
Escalation criteria. Conditions under which the tier moves up.
Communication requirements. Who gets notified, by when.
The table is a reference, not a substitute for judgment. Most incidents fit a pattern in the table; the exceptions get judgment calls.
Counter-argument: "Five tiers is too many; teams will collapse them in practice"
The strongest objection is that five tiers add cognitive overhead that operators will simplify back to two tiers (page or don't) in practice.
The objection is partially right. Teams under pressure do collapse classifications. The defense is to make the tier model integrated into the tooling — the alerting system tags severity automatically based on the trigger criteria, the response runbook routes by tier, the postmortem template differs by tier — so that the tier is a property of the incident handling system, not a discipline the operator has to maintain by remembering. With tooling support, five tiers is sustainable. Without it, two tiers will win regardless of how many you draw.
What Armalo does
The Armalo platform classifies incidents by tier using configurable trigger criteria. Default triggers reflect the patterns above; operators customize for their domain. The platform's alerting routes by tier — S1 pages on-call, S2 sends business-hour alerts, S3 logs to the daily standup, S4 logs to the weekly review, S5 captures to backlog. The postmortem template differs by tier with appropriate depth requirements. Operators who use the defaults get the discipline; operators who customize get the platform behavior they want.
FAQ
What if an incident matches multiple triggers at different tiers? The highest tier wins. Multi-trigger incidents are usually more serious than any single trigger.
How do you handle incidents that span multiple agents or pacts? They are typically S2 or S1 depending on the systemic implications. The response team is sized accordingly.
What about incidents detected by counterparties before the operator's monitoring catches them? Counterparty-reported incidents are usually S2 minimum because the operator's detection failed. The detection failure is itself logged for monitoring improvement.
Can the tier model accommodate domain-specific triggers (e.g., financial agents need different criteria)? Yes. Domain-specific triggers extend the table; the tier semantics remain.
How do you avoid alert fatigue from S2 paging? Tune the S2 triggers carefully; if they fire too often, either the threshold is wrong or the underlying behavior actually warrants S2 attention. Persistent over-alerting is a tooling problem with a tooling fix.
What about incidents in third-party dependencies (model provider outage, settlement chain congestion)? They are tracked as separate incidents with their own severity. Agent-level impact may be S2 or S1 depending on how the dependency failure manifests.
How does the model interact with the operator's broader corporate incident management? It is a specialization of the general model, not a replacement. Major agent S1 incidents typically also trigger the corporate incident process; routine agent operations live in the agent-specific tier model.
Bottom line
Five tiers, integrated into tooling, with specific triggers and response SLAs. The discipline produces faster response to real incidents and less waste on noise. Operators who paged for everything will find their on-call sleeping again; operators who paged for nothing will find they catch incidents at S2 instead of at counterparty-reported S1. Print the table. Tune the triggers. Wire it into the alerting. Then use it consistently.
Continue exploring
Explore more articlesComments
Loading comments…