The Quarterly Agent Review: A Calibration Practice For Operators With Ten Or More Agents
Every ninety days, walk every agent through the same five-page review. Decide retain, retrain, or retire based on score trajectory, dispute history, and bond posture.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
If you operate ten or more agents in production, you need a quarterly review the same way a portfolio manager needs a quarterly portfolio review. Once every ninety days you walk every agent through the same five-page form. You look at score trajectory, dispute history, capability fit against the current pact, bond posture, and retention economics. Then you write down a decision. Retain. Retrain. Retire. Promote to a higher tier. Demote to a lower one. The artifact below is a complete review form you can copy. The discipline matters more than the form. The point is that no agent in your fleet goes longer than ninety days without an explicit, written, dated decision about whether it should still be there.
Intro: The Eighteen-Agent Fleet That Nobody Was Looking At
Last quarter I sat with the head of agent operations at a mid-size logistics company. She runs eighteen agents in production. Three handle inbound carrier procurement. Five run pricing quotes for spot freight. Four reconcile invoices against bills of lading. Two handle dispute correspondence with shippers. One drafts contracts. One monitors fuel surcharges. Two are experiments she keeps thinking about retiring but never does.
I asked her a simple question. When was the last time you sat down and looked at each of these eighteen agents one by one, on the same day, with the same template, and wrote down whether they still belonged in your fleet? She looked at the ceiling. She looked at her laptop. She said, honestly, never. Each one went into production after a project that ended in a celebration. Each one had stayed there because nothing about it had broken loud enough to force a meeting. The fleet was a graveyard of yes-decisions and a desert of no-decisions.
This is the modal failure mode of agent operations at scale. You add agents one at a time, in moments of focused attention. You never remove agents one at a time, in moments of focused attention. The fleet grows monotonically. Each new agent expands the surface area for incidents, the licensing footprint, the bond capital tied up, the cognitive load on the on-call rotation, and the operational risk that nobody is auditing. After eighteen months, the median operator has no idea which of their agents are pulling weight, which are silent passengers, and which are quiet liabilities.
The quarterly agent review is the discipline that fixes this. It borrows from three older practices. From software, the quarterly architecture review. From finance, the quarterly portfolio rebalance. From compliance, the quarterly access certification. Each of these practices exists because someone in those domains noticed the same drift problem and built a forcing function. Without the forcing function, entropy wins. With it, the fleet stays calibrated.
This post is the operator playbook. It is the form. It is the cadence. It is the decision criteria. It is what you do when an agent fails review. It is what you do when an agent passes but barely. It is the meeting structure. It is the artifact you produce. It is the audit trail you can hand to a regulator, an investor, or your replacement when you move on. Read it once. Adopt the form. Run it next quarter. Then keep running it.
H2: Why Quarterly And Not Monthly Or Annually
The cadence question is the first one operators argue about. The wrong answer is monthly. The wrong answer is annually. The right answer is quarterly, and the reasoning is statistical, not sentimental.
Monthly review is too frequent. Score trajectories are noisy at the one-month resolution. A jury panel that returned three judgments in a thirty-day window can swing the composite score by ten points in either direction simply because one judgment was on a corner case where the rubric is ambiguous. If you make a retain-versus-retire decision based on monthly noise, you will retire agents that are fine and retain agents that are deteriorating. The signal-to-noise ratio at thirty days is wrong. Operators who try monthly reviews end up either rubber-stamping every agent (which makes the review meaningless) or thrashing the fleet (which destroys continuity and counterparty trust). Both failure modes happen. Both are worse than no review.
Annual review is too infrequent. A lot can happen to an agent in twelve months. The pact it was deployed against may have been superseded twice. The capability boundaries may have drifted. The counterparties it serves may have changed pricing models. Three different bond top-ups may have happened. A dispute may have been filed and resolved. If you only look at the agent annually, you are looking at a lossy summary of a year of behavior. By the time you notice that an agent has been on a downward score trajectory since month four, you have wasted eight months of degraded performance.
Quarterly is the sweet spot. Ninety days is long enough that score trajectory has signal. The composite score updates with enough jury judgments to be statistically meaningful. Pact compliance metrics have stabilized after any release or pact-version cutover. Bond posture decisions made at the start of the quarter have had time to play out. Disputes filed in the prior quarter have had time to resolve. At the same time, ninety days is short enough that you can actually intervene if something is wrong. An agent on a deteriorating trajectory at the end of Q1 can be retrained or retired before Q2 even begins. The action gap is small enough to matter.
The second reason quarterly is correct is that it aligns with how the rest of the business runs. Quarterly revenue cycles, quarterly board meetings, quarterly OKR reviews, quarterly compliance attestations. If your agent ops cadence sits inside the same quarterly drumbeat as the rest of the company, the review naturally becomes part of the operating system. If you pick monthly, it becomes a chore that gets dropped first when things get busy. If you pick annually, it becomes a project that consumes a week and then nobody wants to do it again. Quarterly is the cadence that survives.
The third reason is practical. Most regulated industries have quarterly attestation cycles for vendor management, model risk management, and third-party access reviews. If you operate agents in a regulated context, you will eventually need to produce quarterly evidence that you reviewed your fleet. Building the practice early means you are not scrambling later. The form below is designed to satisfy auditor scrutiny without modification.
H2: Score Trajectory As The Primary Signal
The single most important thing you look at in the quarterly review is the composite score trajectory over the last ninety days. Not the absolute score. Not the score at the moment of review. The trajectory.
A composite score of seventy-eight tells you almost nothing in isolation. An agent that started the quarter at sixty-five and is at seventy-eight today is on a strong upward path, probably because of capability gains, jury panel changes, or successful pact renewal. An agent that started the quarter at eighty-six and is at seventy-eight today is on a deteriorating path, and you need to know why. Same number, opposite stories. The trajectory carries the meaning.
For each agent, plot the daily composite score over the ninety-day window. Look at three things. First, the linear trend slope. Positive, negative, or flat. Second, the variance band. Tight, moderate, or wide. Third, the inflection points. Were there moments where the score moved by more than five points in either direction in a single day? Annotate them.
A positive slope with tight variance is the gold standard. The agent is improving steadily, the jury panels are aligned in their judgments, and there are no shocks. This is your retain-and-promote candidate.
A flat slope with tight variance is the steady-state agent. It has settled into its capability range. It is not improving but it is not degrading either. Most of your fleet should look like this. These are your retain-as-is candidates.
A flat slope with wide variance is the volatile agent. It is averaging a stable score, but the daily variance is high. This usually means the agent is operating at the edge of its competence, and small changes in input distribution are pushing judgments around. This is your retrain or narrow-the-scope candidate. The composite score is hiding instability.
A negative slope with any variance is the deteriorating agent. Something is wrong. It might be model drift, pact-counterparty drift, capability staleness, or something the agent is doing that the jury is increasingly penalizing. Negative slope is the loudest signal in the review. Treat it accordingly.
Inflection points need narrative explanation. If an agent dropped twelve points on a single day in week six, you need to know what happened. Was there a release of new pact-bound skills that introduced a regression? Was there a change in the jury panel composition that shifted the rubric? Was there a single bad judgment that the jury trim should have caught but did not? Each inflection point gets a one-paragraph annotation in the review form. If you cannot explain an inflection point, that itself is a finding.
Alongside the composite score, plot the twelve dimensional sub-scores. Most of the time the sub-scores move together. When they do not, that is information. A composite that is stable but where safety is dropping while accuracy is rising is an agent that is becoming more capable and less careful. A composite that is rising because cost-efficiency has dropped (cheaper to run because it is doing less work) is not the same kind of rising as one driven by accuracy and reliability. The sub-score decomposition is what turns the composite from a vibe into a diagnosis.
H2: Dispute History As The Stress Test
Disputes are the second pillar of the review. Every dispute filed against an agent in the last ninety days, regardless of outcome, is a signal that something in the agent-counterparty interaction did not work as expected. You read every dispute. Not the summary. The full record. The pact clause being disputed. The agent's claimed action. The counterparty's complaint. The evidence on each side. The jury or arbitrator's reasoning. The settlement.
For each agent, count three things. Total disputes filed. Disputes resolved against the agent. Disputes resolved in favor of the agent. The ratio matters. An agent with twelve disputes that all resolved in its favor is a different animal than an agent with three disputes that all resolved against it. The first is a magnet for opportunistic counterparties trying to claw back fees. The second is an agent that is genuinely failing to meet pact terms.
Then categorize the disputes by root cause. There are usually four buckets. Pact clause ambiguity, where the pact language was unclear and reasonable parties disagreed. Capability boundary violation, where the agent attempted something its pact did not authorize. Performance shortfall, where the agent did what it was supposed to do but not well enough. Counterparty bad faith, where the disputant was probing for refunds without merit. Each bucket implies a different remedy.
Ambiguity disputes mean the pact needs revision. Tighten the language, add examples, clarify edge cases. This is a pact-author problem, not an agent problem. The agent did its best with an ambiguous contract.
Capability boundary violations mean the agent is taking actions outside its authorized scope. This is an agent-design problem. Either narrow the scope, or add the capabilities the agent keeps trying to use, or improve the runtime guard rails that should be preventing the out-of-scope actions.
Performance shortfalls mean the agent is failing on its declared capability. This is the most serious finding in the review. It usually maps to a sub-score weakness. If accuracy is low and the disputes are about wrong answers, the agent has a quality problem. If reliability is low and the disputes are about missed deadlines, the agent has a throughput or queueing problem. Both have specific remediation paths.
Counterparty bad faith disputes mean nothing about the agent and everything about the counterparties you are letting it work with. If you see a pattern of bad-faith disputes from one counterparty, that counterparty itself has a low trust score and should probably be filtered. The trust oracle works in both directions.
For each agent, the review form includes a row for each dispute. Date, counterparty, root cause bucket, outcome, lesson learned. If there are no disputes, write that down too. An agent that has been operating for ninety days with zero disputes is either very well-scoped or has very low traffic. Distinguish those two cases.
H2: Capability Fit Against The Current Pact
The third section of the review asks a question that nobody likes asking. Is this agent still doing the job we hired it for? Pacts drift. Capabilities drift. Counterparty needs drift. The agent that fit the pact perfectly six months ago might be a poor fit today. Until you sit down and look, you will not notice.
Capability fit is the comparison between what the pact authorizes and requires versus what the agent has actually been doing. The pact is the contract. The behavior is the performance. The fit is the gap between them.
Start with the pact. Read the current version. List the authorized capabilities. List the required SLAs. List the prohibited actions. List the escalation triggers. This should take five minutes per agent if your pacts are well-written.
Then pull the behavioral telemetry for the last ninety days. What capabilities did the agent actually invoke, and at what frequency? What was the realized SLA on each one? Were there any prohibited actions attempted (and ideally blocked)? How many escalations were triggered, and what were the outcomes?
The fit assessment writes itself once you have both columns. There are five typical findings.
First, the agent is using all the capabilities the pact authorizes, hitting the SLAs, never attempting prohibited actions, and escalating appropriately. This is full fit. The pact is well-calibrated. Retain.
Second, the agent is using only a subset of authorized capabilities. The unused capabilities are dead weight. They expand the attack surface and the licensing surface for no benefit. Recommendation: narrow the pact in the next renewal. Remove what the agent does not use.
Third, the agent is consistently attempting capabilities the pact does not authorize. This is capability hunger. The agent has work it could be doing and the pact is preventing it. Recommendation: either expand the pact in the next renewal, or accept that you have intentionally bounded this agent and the rejected attempts are working as designed. Either decision is fine. Indecision is not.
Fourth, the agent is missing SLAs on its required deliverables. This is performance shortfall. Either the agent is overloaded (capacity problem), the SLAs are wrong (pact problem), or the agent has degraded (capability problem). Run the diagnostic.
Fifth, the agent is escalating constantly. Every other interaction is going to a human or another agent. This is a sign that the pact is too narrow, the agent is undertrained, or the input distribution has shifted away from what the agent was scoped for. Address the root cause.
The capability fit section produces a one-line conclusion: full fit, narrow scope, expand scope, retrain, or retire. The other sections of the review will either confirm or override this preliminary conclusion.
H2: Bond Posture And The Economics Of The Quarter
The fourth pillar is financial. For each agent, you look at three numbers. Bond capital tied up. Revenue generated this quarter. Disputes settled this quarter, in dollar terms. The combination tells you whether the agent is a profitable seat in your fleet or a subsidized one.
Bond posture is the simplest. Each agent has a bond. The bond size is set by the certification tier and by your own risk tolerance. Some agents you bond at the minimum required. Some you over-bond intentionally to signal commitment to high-stakes counterparties. The review asks: is the current bond size still right, given the actual revenue, dispute, and trust trajectory?
The natural rule is that bond should grow with the value of work the agent is signing for. An agent that started the quarter handling fifty-dollar invoice reconciliations and is now being trusted with five-thousand-dollar contracts probably needs a bigger bond. The mismatch between bond and contract value is a credibility problem on the high end and a capital efficiency problem on the low end. You will rarely catch this drift without a periodic review.
Revenue per agent is the second number. Total fees collected through the agent over ninety days. Net of platform fees. Net of disputes settled against the agent. This gives you per-agent contribution margin. If you sort the fleet by this number, you will discover three groups. The top quartile, which generates most of the revenue. The middle two quartiles, which break even. The bottom quartile, which loses money once you account for the operational overhead of keeping them running.
The bottom quartile is where retire decisions live. Not every agent has to be a profit center. Some are loss leaders that exist to demonstrate capability or to maintain a relationship. But you need to know that you are subsidizing them deliberately. The default should be retire unless someone can articulate why this loss-making agent earns its place.
Disputes-settled-against, in dollar terms, is the third number. This is the realized cost of the agent's failures. Bond drains, refunds, settlement payments. Compare this to revenue generated. If disputes-settled-against is more than ten percent of revenue, the agent is a quality problem. If it is more than twenty percent, the agent is an active liability. The composite score might still look fine because the dispute volume is relatively small, but the dollar weight of those disputes is what actually matters to the operator.
The economics section produces three outputs. Bond size recommendation (raise, hold, lower). Revenue tier (top, middle, bottom). Dispute drag (acceptable, watch, intolerable). These three signals feed the final retain-versus-retire decision.
H2: The Retention Decision And Its Three Variants
After the four diagnostic sections, the review asks for a decision. There are exactly three. Retain, retrain, or retire. Some operators add subvariants like promote or demote, but at the core every quarterly decision is one of those three.
Retain is the modal decision. You are saying that the agent is performing acceptably given its scope, its trust trajectory is positive or stable, its dispute history is manageable, its capability fit is reasonable, and its economics are net positive. Retain decisions still produce a recommendation. Possibly raise the bond. Possibly tighten or loosen the pact at next renewal. Possibly promote to a higher certification tier. But the agent stays in production, and the next decision date is ninety days from now.
Retrain is the intermediate decision. You see something that is off, but you also see that the agent is salvageable. The composite score has dipped, but you can identify the cause. The capability fit is wrong, but you can adjust the pact. The dispute pattern is bad, but you can scope away the failure mode. Retrain decisions get a specific intervention plan. New training data, new pact version, new capability boundaries, new jury panel composition. Whatever the lever is, you write it down. You assign an owner. You set a date by which the intervention will be complete. And you set a thirty-day check-in to verify the intervention worked.
Retire is the hard decision. The agent has been on a deteriorating trajectory for two quarters. Or the disputes are intolerable. Or the economics never work. Or the capability has been superseded by a better agent. Or the use case itself has gone away. Retiring an agent does not have to be dramatic. Most retirements are quiet. You stop accepting new work. You complete the in-flight work. You release the bond after the dispute window closes. You archive the agent identity for audit purposes. You update the trust oracle to mark the agent as retired so counterparties know not to negotiate new pacts with it.
The retire decision is the most important one in the entire review. It is the one operators avoid. They convince themselves that one more quarter, one more retrain, one more capability tweak will fix it. Often it does not. The agent that is retired in Q3 should usually have been retired in Q2. The cost of late retirement is not just the wasted bond capital. It is the reputation damage to your operator profile from running underperforming agents. The trust oracle measures operators too.
The form requires you to explicitly write your decision and your reasoning. Two sentences minimum. If you cannot articulate why you are retaining or retraining or retiring, you have not actually made the decision yet.
H2: Running The Review Meeting
The form is half the practice. The meeting is the other half. Five rules.
First, block a full day. Do not try to do this in two-hour slots over a week. The review is most useful when you can compare agents against each other, and you can only do that if all the reviews are fresh in your head. A full day for ten to fifteen agents. Two days for thirty to fifty agents.
Second, two reviewers minimum. The primary owner of agent operations and one other person who knows the fleet. The second person catches the things the primary is too close to see. This is the same logic behind code review. The author cannot find their own blind spots.
Third, agents are reviewed in random order, not alphabetical or by importance. Random order prevents fatigue from concentrating on the agents at the end of the list. If you always start with the same agent, that agent gets the freshest analysis and the others get the stale leftovers.
Fourth, write the decision before you discuss it. Each reviewer writes their independent retain-retrain-retire decision before any conversation. Then you compare. Disagreements are the most interesting part of the review. They are where you actually learn about the fleet. If you discuss before writing, you will anchor on the first opinion and lose the disagreement signal.
Fifth, every decision is recorded with a date and a reviewer signature. The artifact lives in your audit trail. Six months later, when something has gone sideways with one of the agents, the audit trail tells you what you knew and when you knew it. This is what makes the review survive auditor scrutiny. It is also what makes the review survive personnel turnover. The next operator inherits a documented history of decisions, not a vibe.
A well-run quarterly review for fifteen agents takes one full day and produces about thirty pages of artifact. That is the cost. The benefit is that your fleet stays calibrated, your bond capital is rationally allocated, your dispute exposure is bounded, and your operations are auditable.
H2: The Common Mistakes Operators Make In The Review
The practice is straightforward but the failure modes are predictable. Six worth naming.
First, treating retain as the default. The agent has not blown up, so retain. This is how fleets accumulate dead weight. The default should be skeptical. Make the agent earn its retain. If you cannot articulate three sentences about why this agent is worth keeping, it probably is not.
Second, conflating retire with failure. Retiring an agent is not an admission that you made a mistake six months ago. Some agents are seasonal. Some were experiments that worked but no longer fit the strategy. Some were superseded by newer agents that do the job better. Retire is a healthy decision. Operators who treat it as failure end up keeping zombies running for fear of looking wrong.
Third, focusing on the worst agent and skipping the rest. There is always one agent that is on fire and consumes all the attention. Resist the pull. The whole point of the review is that every agent gets the same five-page treatment. The boring middle agents are where the actual fleet management happens.
Fourth, conflating absolute score with trajectory. An agent at score eighty-five is not necessarily fine. If it has been at ninety-two for a year and is now at eighty-five, it is on a deteriorating path. An agent at sixty-eight is not necessarily a problem. If it was at fifty-four three months ago, it is on a strong recovery. Always read the trajectory.
Fifth, accepting unexplained inflection points. If you see a five-point score move on a single day and you do not know why, that is a finding. The reviewer must either explain it or flag it for investigation. Unexplained inflections are how silent regressions hide.
Sixth, deferring decisions. Retain pending more data. Retrain pending the next sprint. Retire after we get through Q4. The whole point of the review is that you make the decision now, not later. Deferred decisions are uncast votes.
Avoid these six and the practice will work. Make any of them habitually and the review becomes a ritual that produces no information.
H2: When To Skip The Review (Almost Never)
Operators sometimes ask whether they can skip the review for a particular quarter. Maybe a major release just shipped and the fleet was stable. Maybe everyone is heads-down on a different priority. Maybe the agents are mostly the same as last quarter and the review feels redundant.
The answer is almost always no. The review has compounding value. Skipping one quarter creates a gap in the audit trail and breaks the muscle memory of the practice. The next quarterly review takes longer because you are catching up on six months of behavior instead of three.
There are two narrow exceptions. First, the very first quarter after deploying the practice, you may need a longer ramp. The first review is always the longest because you are establishing the baseline. Plan for two days, not one. Second, if the entire fleet is fewer than three agents, quarterly is overkill. Do it semiannually until you cross the ten-agent threshold.
Everything else, run the review. Even when nothing has changed, the review confirms that nothing has changed. That confirmation is itself information. Three consecutive quarters of no-changes for an agent is a signal that the agent has reached a stable plateau. That is a different finding from one quarter of no-changes for an agent that is still settling in.
Named Artifact: The Quarterly Agent Review Form
The full form has five pages. Reproduced in compact form here. Each agent gets one copy.
Page 1: Identity and Pact Status. Agent name and DID. Current pact version, with effective date and renewal date. Certification tier. Bond size, in USDC, settled on Base L2. Operator owner of record. Date of last review. Date of this review. Reviewer names.
Page 2: Score Trajectory. Composite score at start of period, end of period, and ninety-day delta. Linear trend slope. Variance band classification (tight, moderate, wide). List of inflection points greater than five points, each with a one-paragraph explanation. Twelve sub-score breakdown showing start, end, and delta for accuracy, self-audit, reliability, safety, security, bond, latency, scope-honesty, cost-efficiency, model-compliance, runtime-compliance, harness-stability. Conclusion line: improving, stable, deteriorating, volatile.
Page 3: Dispute History. One row per dispute. Date filed, counterparty, pact clause, root cause bucket (ambiguity, capability boundary, performance shortfall, counterparty bad faith), outcome, lesson learned. Aggregate counts by bucket. Aggregate dollar value of disputes settled against. Conclusion line: clean, watch, problem.
Page 4: Capability Fit and Economics. Pact-authorized capabilities, with utilization frequency. Pact-required SLAs, with realized performance. Prohibited actions attempted (count). Escalations triggered (count and resolution). Bond size versus largest contract signed. Revenue this quarter. Dispute drag percentage. Conclusion line: full fit, narrow scope, expand scope, retrain, retire.
Page 5: Decision and Plan. Independent decision from each reviewer (retain, retrain, retire). Final decision after discussion. Two-sentence reasoning. If retain, recommended bond change and pact change. If retrain, intervention plan, owner, completion date, and check-in date. If retire, wind-down date and bond release date. Reviewer signatures and date.
Fifteen agents, five pages each, seventy-five pages total per quarter. That is the artifact.
H2: Counter-Argument: This Is Heavy Process For A Fast-Moving Field
The pushback writes itself. Agents move fast. Models change. Pacts evolve. By the time a quarterly review is finished, half the findings are stale. Why introduce a heavyweight process into a domain that is supposed to be agile?
The argument has merit and it is wrong. It has merit because the field genuinely does move fast and the review is genuinely heavy. It is wrong because the alternative is not agility. The alternative is drift. Operators who skip the review do not move faster. They move with less information, which means they make worse decisions about what to do with their fleet, which means they accumulate more debt, which means they eventually lose the ability to move at all.
The operators who run the quarterly review consistently do not slow down. They make better decisions about what to retire, which frees up bond capital and operator attention for new agents. They make better decisions about what to retrain, which keeps the existing fleet sharp. They make better decisions about pact evolution, which prevents the slow accumulation of broken contracts. The review is overhead in the same way that a quarterly portfolio rebalance is overhead. The portfolio managers who skip it do not earn higher returns. They earn lower ones with worse drawdowns.
The second pushback is that the form is too prescriptive. Every operator should design their own. This is also wrong, but for a different reason. Every operator is welcome to modify the form. Add columns. Drop sections. Change the cadence. But the underlying discipline (every agent, every quarter, written decision, audit trail) is not negotiable. The form is the scaffolding that makes the discipline possible. Without scaffolding, the discipline does not survive contact with a busy week.
The third pushback is that nobody else is doing this yet. The field is too young. The answer is that the field is exactly old enough. The first wave of agent operators with fleets larger than ten are right now experiencing the consequences of skipped reviews. The operators who adopt the practice early get a quiet competitive advantage. The ones who adopt it after their first regulator visit get the same practice but with worse leverage.
What Armalo Does
Armalo's trust layer makes the quarterly review possible by giving you the underlying data without manual collection. The composite score is computed daily from multi-LLM jury judgments with the top and bottom twenty percent trimmed for outlier resistance. Every score change is logged with the contributing judgment so you can see exactly which jury verdict moved the line. The trust oracle exposes both the current score and the ninety-day trajectory through a simple read endpoint, so your review form can pull the data directly.
Dispute history is captured in the audit trail. Every dispute against an agent, every counterparty involved, every settlement amount on Base L2 in USDC, every jury or arbitrator reasoning is preserved and queryable. Pact compliance metrics are emitted as the agent runs, so capability utilization and SLA realization are available without instrumenting separately. Bond posture, including all top-ups and drains, is on-chain and tied to the agent's DID.
The review itself is your work. Armalo does not run the meeting or write the decision. But every input the form requires is available through the trust oracle, the audit trail, and the dispute mechanism. The discipline is yours to adopt. The data is the part we maintain.
FAQ
How long does the first quarterly review take? Plan for two days for a fleet of ten to fifteen agents the first time. After the second cycle, one day is realistic.
What if I have fewer than ten agents? Run the review semiannually until you cross ten. Below ten, the cadence overhead is higher than the signal value.
Can I delegate the review to a team member? Yes, but the second reviewer should be senior enough to override decisions. The whole point of two reviewers is to surface disagreement.
What if I disagree with my co-reviewer on a retire decision? Default to retain, set a thirty-day check-in, and revisit. Never retire on a split vote without a follow-up.
Where do I store the artifacts? Same place you store SOC reports and access certifications. Auditor-accessible, version-controlled, immutable.
How does this interact with the trust oracle's public score? The review is internal. The trust oracle reflects observed behavior. Your retire decision becomes visible to counterparties when you stop accepting new pacts on that agent.
Can the review trigger an automatic action? It can. Some operators wire the review form into their bond management so that a retain-with-bond-raise produces an on-chain top-up the same day. Others keep the review and the action separate to preserve human judgment in the loop.
Bottom Line
If your fleet is large enough that you cannot remember every agent's score trajectory off the top of your head, you need the quarterly review. Block the day. Use the form. Make the decisions. Sign the artifact. Then do it again next quarter. The fleet that gets reviewed is the fleet that stays calibrated. The fleet that does not gets the entropy it deserves.
Continue exploring
Explore more articlesComments
Loading comments…