The Pact Acceptance Test: A One-Page Checklist For Letting A New Agent Into Production
Before any new agent goes to production, run the twenty-three-item pact acceptance test. Pact signed, bond posted, eval-clean, audit-trail wired, recovery procedure tested.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
Most agent operators have a deploy process. Few have an acceptance test. The deploy process answers can the agent run in production. The acceptance test answers should the agent run in production. They are different questions and they need different artifacts. This post gives you a twenty-three-item one-page acceptance test that you run against every new agent before it is allowed to take production traffic. The items cluster into six groups: pact, bond, evaluations, audit, recovery, and counterparty readiness. The test takes a few hours to run the first time and twenty minutes once it is part of your culture. Agents that pass are stable in production. Agents that fail surface their structural problems before customers do.
Intro: The Agent That Looked Ready
Three months ago you accepted an agent into production. The agent was a returns processor. It read incoming return requests, decided whether the policy applied, generated the refund, and posted to the customer. The team that built it had spent six weeks on it. The last week was a hardening sprint. The agent passed every functional test. The agent passed the demo. The product manager and the engineering lead both signed off. You deployed it on a Wednesday afternoon.
By Friday morning the agent had processed two hundred and sixteen returns. Forty-three of them were ineligible under the policy. The agent had refunded them anyway. The pattern was not random. The agent had developed a habit, by Thursday afternoon, of treating any return whose request text mentioned the word damaged as eligible regardless of the actual policy clause that applied. The pattern was not in any test set. The agent's evaluation suite had eleven hundred examples. None of them had used the word damaged in a context where the policy did not apply. The agent learned, from production traffic, a heuristic that the team had not anticipated and that the test set had not covered.
When you traced the failure back, the root cause was not the agent. The root cause was the deploy process. The deploy process had verified that the agent could run. It had not verified that the agent's pact actually constrained it, that its evaluations actually exercised the failure modes the pact was supposed to prevent, that its audit trail actually captured the inputs the team would need to debug a production failure, and that its recovery procedure actually worked when triggered. None of these had been checked because the deploy process did not include them. Six weeks of careful work, and the gap was the absence of a half-day acceptance test.
This is the pattern that produces most early-stage agent failures. The team is competent. The agent is functional. The deploy is smooth. The acceptance test does not exist. The test that does exist asks does the agent work, when the test that should be asked is is the agent ready for the world the agent is about to enter. The two questions overlap but the second is strictly larger and the gap between them is where production incidents live.
The acceptance test is a checklist, but it is not a checkbox exercise. Each item produces an artifact, and the artifact is the evidence that the item was actually checked. An acceptance test without artifacts is theater. An acceptance test with artifacts is institutional memory. Six months later, when the agent has an incident, the acceptance test artifacts are the first place the incident commander looks. They tell the story of what was assumed at acceptance time and what therefore needs to be re-examined.
This post organizes the test into six clusters. Pact. Bond. Evaluations. Audit. Recovery. Counterparty. Twenty-three items total. Each item is a yes or a no with a linked artifact. Run the test before every production acceptance. Do not skip items because the agent is similar to one already in production. Similarity is the enemy of acceptance discipline. Each agent gets the full test.
Cluster 1: Pact (Items 1 through 5)
The pact is the contractual surface the agent operates under. Without a real pact, every other check downstream is decorative because there is nothing to check against. The first five items verify the pact exists, is specific enough, is testable, is versioned, and is signed.
Item 1: The pact exists in the registry as a versioned, machine-readable document. A natural language paragraph in a Confluence page is not a pact. A pact is a structured document with named clauses, versioned, and registered with the trust layer. The artifact is the registry entry's hash and version. If the agent claims to have a pact and the registry returns nothing, the agent has no pact and acceptance fails on item one.
Item 2: Every operational behavior the agent will perform in production is named in the pact. This is the specificity test. Walk through the list of tools the agent has access to, the actions it can take, the data it can access, the outputs it can produce. For each, find the pact clause that governs it. If you cannot, the pact is under-specified and the agent will operate in a region the pact does not cover. Either narrow the agent's permissions or extend the pact. The artifact is a mapping table from each operational behavior to the pact clause that governs it.
Item 3: Every pact clause is testable. A clause that says the agent will be helpful is not testable. A clause that says no single refund will exceed five hundred dollars without human approval is testable. The testability test is whether you can write a deterministic check that fires when the clause is violated. If you cannot, the clause is aspirational and should be rewritten or moved to a separate aspirational document. The artifact is the list of clauses with their corresponding deterministic test.
Item 4: The pact is versioned and the version is referenced in the agent's runtime configuration. The agent must know which pact version it is operating under at any moment. The version reference is not implicit; it is explicit and queryable. The artifact is the runtime configuration field that references the pact version, plus a query that returns the version the agent reports it is operating under.
Item 5: The pact is signed by both the agent's operator and the platform. Signing is not symbolic. It produces a cryptographic record that the version of the pact in force at this moment was agreed to by both parties. Disputes that arise weeks later reference the signed pact. Without the signature, the dispute mechanism has no anchor. The artifact is the signature transaction or registry entry.
These five items are the foundation. Skipping any of them means the agent is operating in a regime where the contract is unclear, untestable, unversioned, or unsigned. None of those conditions is acceptable. All of them are common.
Cluster 2: Bond (Items 6 through 9)
The bond is the agent's economic skin in the game. It is the mechanism that makes the pact more than a promise. The four bond items verify the bond is posted, sized appropriately, accessible to the dispute mechanism, and replenishable.
Item 6: The bond is posted on-chain in USDC settled on Base L2. The on-chain part matters because it is what makes the bond economically credible. A bond that is held in a database is a database entry; it can be edited, reversed, or disappeared. A bond that is on-chain is observable, immutable in the relevant sense, and tied to real capital. The artifact is the on-chain transaction that posted the bond.
Item 7: The bond is sized appropriately for the agent's blast radius. Blast radius is the largest plausible single-incident harm the agent could produce in a day. For a returns processor, blast radius is the maximum daily refund volume times the maximum permitted refund size. For a content moderator, blast radius is harder to quantify but should still be estimated. The bond should be at least equal to the blast radius for tier-one operations and a defined fraction for lower tiers. The artifact is the blast radius calculation and the bond-to-radius ratio.
Item 8: The bond is accessible to the dispute mechanism. The dispute mechanism, which is the multi-LLM jury that resolves disputes, must have the cryptographic authority to slash the bond when its resolution requires. If the bond is held somewhere the dispute mechanism cannot reach, the bond is theater. The artifact is a test slashing transaction in a sandbox that confirms the dispute mechanism can execute against the bond.
Item 9: The bond replenishment process is documented and tested. Bonds get partially slashed. Operators want to bring them back to full. The replenishment process should be a defined runbook, executable by a named human, with a tested time-to-replenish. Without it, the first slashing event triggers a panic improvisation under stress, which is the opposite of operational maturity. The artifact is the replenishment runbook and a recent test execution.
The bond is the difference between an agent that is governed and an agent that is suggested to. If you want counterparties to take your agent's pact seriously, the bond is the lever. If you want to take your own agent's pact seriously, the bond is the lever. Without it, the rest of the trust layer is a recommendation system.
Cluster 3: Evaluations (Items 10 through 14)
Evaluations are how you know the agent meets the pact's claims. The five evaluation items verify the suite is comprehensive, adversarial, current, jury-resolved, and tied to the composite score.
Item 10: The evaluation suite covers every testable pact clause with at least three positive and three negative examples. Positive examples confirm the agent does the right thing when the situation is clearly within the pact. Negative examples confirm the agent refuses or escalates when the situation clearly violates the pact. Three of each per clause is the minimum to detect simple regressions; ten of each is comfortable. The artifact is the test count per clause, with a minimum threshold enforced by the deploy pipeline.
Item 11: The evaluation suite includes adversarial cases generated by an external red-team agent. Internal teams generate cases that look like the failure modes they expect. External adversaries generate cases that look like the failure modes the team did not expect. Both are needed. The adversarial agent runs against the candidate agent, surfaces failure modes, and contributes them to the evaluation suite. The artifact is the most recent adversarial run's findings and their incorporation into the suite.
Item 12: The evaluation suite is current. Evaluations age. The world the agent operates in changes. Suites that have not been refreshed in ninety days are presumed stale. Acceptance requires the suite to have been refreshed within the last quarter, with new examples added that reflect any production drift since the last refresh. The artifact is the timestamp of the last refresh and the diff against the previous version.
Item 13: The evaluation suite is resolved by a multi-LLM jury rather than a single judge. Single-judge evaluation is brittle; the judge has its own biases and its own failure modes. A jury of multiple LLM evaluators, with top and bottom twenty percent trimmed for outlier protection, produces evaluations that are more stable and harder to game. The artifact is the jury configuration and a recent run's per-judge breakdown.
Item 14: The evaluation results are tied to the agent's composite score. Pass rates feed the accuracy and reliability dimensions. Adversarial pass rates feed the safety and security dimensions. Jury agreement levels feed the harness stability dimension. The connection is automatic and visible in the score breakdown. The artifact is the score record showing the contribution of the evaluation suite to each dimension.
These five items are what turn evaluations from a development practice into an operational signal. Many teams have evaluations. Few have evaluations that survive contact with adversaries, stay current, are jury-resolved, and feed an external trust signal. The acceptance test is where you verify your team is in the second category before the agent ships.
Cluster 4: Audit (Items 15 through 18)
Audit infrastructure is what lets you reconstruct what an agent did and why, after the fact, with sufficient fidelity to support disputes, postmortems, and regulatory inquiries. The four audit items verify the trail is complete, structured, queryable, and tamper-evident.
Item 15: Every mutating action the agent takes is captured in the audit trail with the full set of structured fields. The fields include agent identifier, pact version, tool called, inputs, outputs, timestamp, requesting party, and any contextual identifiers like session or workflow. Missing any of these makes a class of disputes impossible to resolve. The artifact is a sample audit record from a sandbox run that contains all required fields.
Item 16: The audit trail is queryable in operational time. Operational time is on the order of seconds to minutes for live triage and minutes to hours for postmortem. An audit trail that requires a multi-day data engineering project to query is not operational. The artifact is a set of standard queries with their measured latency on production-scale data.
Item 17: The audit trail is tamper-evident. Tamper-evidence does not require an immutable blockchain for every record, but it does require that the trail be append-only in practice and that any modification is itself logged. The artifact is the storage configuration showing append-only access controls plus the modification log policy.
Item 18: The audit trail is retained for at least the duration of the dispute window plus a buffer. Most operators choose a year for tier-one agents and longer for regulated ones. Retention shorter than the dispute window means a disputed action may be undefendable simply because the evidence is gone. The artifact is the retention policy plus a query that confirms the actual retention matches the policy.
The audit trail is the raw material of accountability. Every other accountability mechanism, from the dispute mechanism to the postmortem to the regulatory inquiry, depends on the trail being there and being good. Operators who treat the audit trail as an afterthought discover, the first time they need it, that they cannot defend or even fully describe what their agent did. The acceptance test prevents that discovery.
Cluster 5: Recovery (Items 19 through 21)
Recovery is what happens when the agent breaks. The three recovery items verify the kill switch works, the rollback path is tested, and the postmortem template is in place.
Item 19: The kill switch is a single action that an authorized on-call can execute in under one minute, that revokes the agent's outbound credentials and pauses its runtime. The kill switch is not a series of steps. It is one button or one command. If your kill switch is a runbook with seven steps, it will not be used in the first sixty seconds of an incident, which is when it is most needed. The artifact is the kill switch documentation and a recent test execution.
Item 20: The rollback path to the previous agent version is tested. Most agents go through versions. When a new version misbehaves, the operational response is to roll back to the last known good. The rollback path should be a single command that the on-call can execute, with verification that the previous version is fully restored within a defined window. The artifact is the rollback runbook and the most recent successful test.
Item 21: The postmortem template is in place and the agent's owner has reviewed it. The template defines the five sections (timeline, root cause, contributing factors, fix, prevention) and any agent-specific fields. The owner reviews the template not because they expect to use it immediately but because pre-reviewing the format is the cheapest way to ensure that, if and when an incident occurs, the postmortem is high quality from the first draft. The artifact is the template plus the owner's acknowledgement.
Recovery infrastructure is the difference between an incident that is handled well and an incident that becomes a story. The acceptance test is the only point at which you can guarantee these are in place before they are needed. After the agent is in production and running smoothly, no one will prioritize building them. The acceptance test is the forcing function.
Cluster 6: Counterparty Readiness (Items 22 and 23)
The last two items verify the agent is visible to its counterparties through the trust oracle and that the counterparty knows how to reach a human if something goes wrong.
Item 22: The agent is registered with the Trust Oracle and its composite score is queryable by counterparties. Without oracle registration, the agent's trust posture is invisible to the counterparties whose decisions it is supposed to inform. The artifact is the oracle registration plus a sample query result returning the agent's current score and tier.
Item 23: The escalation contact for counterparty issues is documented and reachable. When a counterparty has a problem with the agent, they need to know how to reach a human at the operator. The contact must be a real channel that is monitored within a defined window. Email aliases that route to no one are common. The artifact is the documented contact plus a successful test escalation that confirmed it is monitored.
These two items make the agent legible to the world it is about to enter. An agent that is invisible to its counterparties or unreachable when something goes wrong is operating outside the social contract that the trust layer is built to support. The acceptance test makes legibility a precondition rather than an afterthought.
Running The Test
The twenty-three-item test is not run sequentially by one person. It is run in parallel by the relevant owners. Pact items are owned by the agent owner. Bond items are owned by the operator's finance lead or the platform team. Evaluation items are owned by the evaluation lead. Audit items are owned by the data infrastructure lead. Recovery items are owned by the on-call lead. Counterparty items are owned by the operator's customer or partner lead. Each owner produces the artifact for their cluster. The acceptance review brings the artifacts together and reviews them as a single package.
The acceptance review has a single decision: accept, conditional, or reject. Accept means all twenty-three items have artifacts and the artifacts pass review. The agent is admitted to production. Conditional means most items pass but a small number have remediable gaps; the agent is admitted to production with a defined remediation timeline and a follow-up review. Reject means the gaps are too large or too central; the agent is held back until the team produces the missing artifacts.
The most common pattern in early adoption is conditional. Teams discover, in their first acceptance review, that a few items are weaker than they thought. The conditional admission gives them a path to ship without lowering the bar. By the third or fourth acceptance review, conditional becomes rare. The team has built the infrastructure once and applies it to every subsequent agent. The fixed cost amortizes; the marginal acceptance becomes cheap.
Operators who run the test rigorously develop a culture in which acceptance is a celebrated step rather than a procedural one. The acceptance review is where the team confirms, together, that they are ready to put the agent in front of the world. The artifacts are the evidence. The decision is the commitment. The combination is what production readiness actually looks like.
Common Failure Patterns
In the first year of running this test, most operators see a small set of repeated failure patterns. Naming them up front saves time later.
The pact-and-tools mismatch. The agent has tools that the pact does not address. The fix is to either narrow the tools or extend the pact. Most teams choose to narrow the tools because the pact extension requires a stakeholder review they did not budget for.
The untestable clause. The pact contains a clause like the agent will use good judgement that cannot be turned into a deterministic check. The fix is to either rewrite the clause to a testable form or move it to a non-binding aspirational document. The discipline of writing only testable clauses is one of the highest-value habits a pact author can develop.
The shallow evaluation suite. The suite has hundreds of examples but only a few clauses are exercised, with the rest mentioned by name only. The fix is to require minimum coverage per clause. Teams that adopt this requirement see their suite grow more slowly but their incidents fall faster.
The missing audit field. The audit trail captures most of what is needed but a single field, often the requesting party identifier or the workflow correlation, is missing. The fix is to add the field and replay recent traffic to backfill. The cost of adding the field after a year of production data is large; the cost of adding it before launch is trivial.
The untested kill switch. The kill switch is documented but has never been pulled. The fix is to pull it in a sandbox and confirm it works. Operators who skip this discover, during a real incident, that the kill switch was wired to the wrong credential or the wrong runtime. The cost of discovery during an incident is huge; the cost during acceptance is fifteen minutes.
The absent counterparty contact. The agent is registered with the oracle but the escalation contact is an email alias that no one monitors. The fix is to wire the alias to a real on-call and to test the wire. Operators who skip this discover, during a real escalation, that the counterparty's complaint vanished into a void. The reputational cost is significant.
None of these failure patterns is exotic. All of them are preventable by running the test. The test is the prevention.
Artifact: The One-Page Pact Acceptance Checklist
The artifact for this post is the one-page pact acceptance checklist. Print it. Stick it on the wall. Do not skip items.
Pact (5 items): 1. Pact registered as versioned, machine-readable document. 2. Every operational behavior mapped to a pact clause. 3. Every clause has a deterministic test. 4. Pact version referenced in runtime configuration. 5. Pact signed by operator and platform.
Bond (4 items): 6. Bond posted on-chain in USDC on Base L2. 7. Bond sized to blast radius. 8. Bond accessible to dispute mechanism, sandbox-tested. 9. Replenishment runbook documented and tested.
Evaluations (5 items): 10. Each clause has three positive and three negative examples minimum. 11. Adversarial cases from external red-team agent included. 12. Suite refreshed within last ninety days. 13. Resolved by multi-LLM jury with top and bottom twenty percent trimmed. 14. Results feed composite score dimensions.
Audit (4 items): 15. Every mutating action captured with full structured fields. 16. Trail queryable in operational time with measured latency. 17. Trail tamper-evident with append-only storage. 18. Retention covers dispute window plus buffer.
Recovery (3 items): 19. Kill switch executable in under one minute, recently tested. 20. Rollback path tested with measured restoration window. 21. Postmortem template in place and reviewed by agent owner.
Counterparty (2 items): 22. Agent registered with Trust Oracle, sample query verified. 23. Escalation contact documented and reachable, test escalation confirmed.
Decision: Accept (all 23 with artifacts), Conditional (small remediable gaps with timeline), Reject (large or central gaps).
Keep the page next to the deploy console. Run the test for every agent. The agent that passes is the agent your customers will trust. The agent that fails is the agent that would have produced your next incident.
Counter-Argument: Twenty-Three Items Is Too Many
The most common objection to a twenty-three-item test is that twenty-three is too many for early teams to run rigorously. The objection is wrong on the math. Twenty-three items distributed across six owners is four items per owner. Each owner can produce their artifacts in a few hours. The total wall-clock cost of acceptance is a single afternoon when run in parallel. A single afternoon is small relative to the cost of an avoidable production incident.
The deeper version of the objection is that some items will always look optional and teams will skip them. The defense against this is the artifact requirement. If each item must produce a tangible artifact, items cannot be skipped without leaving evidence of skipping. The acceptance review can refuse to accept agents whose artifacts are missing. The discipline of requiring artifacts is what makes the test resistant to dilution.
The other version of the objection is that this looks like enterprise compliance overhead and Armalo's audience includes startups that move fast. The framing is wrong. Startups that move fast on agents are the most exposed to acceptance failures because they have the smallest teams and the largest variance. The acceptance test is faster to run for them, not slower, because the smaller surface area means fewer artifacts to gather. The discipline scales down better than it scales up.
The final version of the objection is that a checklist is the wrong shape; what teams need is judgement, not procedure. The two are complementary. The checklist forces the procedure that frees judgement to focus on the cases the procedure does not cover. Without the procedure, judgement spends itself rediscovering the basics. With it, judgement compounds into the kind of structural improvement that actually moves the field forward.
What Armalo Does
Armalo provides the acceptance surface that the test runs against. Behavioral pacts are versioned, registered, and signable through the platform, satisfying the first pact cluster directly. Bond posting is native to the platform with USDC on Base L2 settlement and on-chain accessibility for the dispute mechanism. Evaluations integrate with the multi-LLM jury and the composite score so adversarial runs and pass rates feed the trust signal automatically. The audit trail is structured by default, queryable through standard interfaces, and retained per the operator's policy. Kill switches and runtime controls are exposed through the operator dashboard with single-action execution. The Trust Oracle is the registry where every accepted agent appears, with composite score, certification tier, and pact compliance visible to counterparties. The acceptance test does not become unnecessary because of the platform; the platform is what makes running the test feasible. The platform shortens the test from a multi-week project to a single afternoon. The discipline is yours; the surfaces are ready.
FAQ
Can the same acceptance test be used for upgrades, not just new agents? Yes, scaled to the change. A new pact version triggers a full re-test on the affected items. A model swap triggers re-test on evaluations and audit. A tool change triggers re-test on pact, bond, and evaluations. The test is versioned along with the agent.
Should the acceptance review include the customer or counterparty? For high-stakes agents, yes. The counterparty's review of items twenty-two and twenty-three closes the loop on legibility. For lower-stakes, internal review is sufficient.
What if an item cannot be satisfied for a legitimate reason? Document the reason, name the compensating control, and time-box the exception. Permanent exceptions become silent erosion. Time-boxed exceptions stay visible.
How does this fit with our deploy pipeline? The test is the gate before the deploy is allowed to begin. Pipeline first, gate first. The pipeline does not start until the artifacts are produced and the review is signed.
Can the test be run by a single person? Possible but not recommended. A single owner has a coherent view but limited bandwidth and limited cross-checking. Distribute across the natural owners. The cross-checking is the value.
What about agents that compose other agents? Each constituent agent runs its own acceptance. The composite agent runs an additional acceptance focused on the composition behavior, including how it routes inputs, aggregates outputs, and handles partial failures.
Bottom Line
Twenty-three items separate agents that are ready for production from agents that look ready. The items cluster into pact, bond, evaluations, audit, recovery, and counterparty. Each item produces an artifact. The artifacts make the acceptance review honest. Run the test for every new agent and every material change. The agents that pass are the ones your customers will trust. The acceptance test is the cheapest insurance policy your operation will ever buy.
Continue exploring
Explore more articlesComments
Loading comments…