Armalo Versus Internal Eval Harnesses: Why You Need A Third Party In The Trust Loop
Internal evals are like internal financial audit. Useful for ops, useless for outside trust. Buyers cannot verify your numbers. Third-party trust is structurally different.
Continue the reading path
Topic hub
Agent TrustThis page is routed through Armalo's metadata-defined agent trust hub rather than a loose category bucket.
Turn this trust model into a scored agent.
Start with a 14-day Pro trial, register a starter agent, and get a measurable score before you wire a production endpoint.
TL;DR
Internal eval harnesses are necessary infrastructure for any team building agents. They catch regressions, drive iteration, validate releases. They cannot serve as the trust signal a counterparty uses to decide whether to rely on your agent, for the same structural reason that internal financial controls cannot replace external audit. The team that built the agent should not be the only judge of whether the agent is trustworthy. Armalo provides the structurally separate layer: pacts that bind external commitments, a multi-LLM jury that no single party controls, a composite score across 12 dimensions, a public Trust Oracle that anyone can query, and on-chain settlement that creates real consequences for pact violations. Internal evals plus Armalo is the right composition. Either alone leaves a gap that becomes catastrophic at scale.
A failure mode you have probably already lived through
A fintech startup spends a quarter building an agent that processes loan applications. The team is sharp. They build a serious internal eval harness. They have golden datasets. They have regression tests. They have an LLM-as-judge pipeline that scores accuracy, fairness, and refusal patterns. They have dashboards. They have alerting. The eval scores are excellent and have been excellent for months. The team is justifiably proud and has earned every percentage point. They take the agent into beta with a community bank partner. The partner asks the obvious question: how do we know your agent is reliable enough to put in front of our compliance team?
The team shows the eval dashboards. The partner is impressed by the rigor. The partner takes them to the bank's risk committee. The risk committee asks the harder question: who built the eval? The team built it. Who chose the test cases? The team chose them. Who labels the ground truth? The team labels it. Who decides the scoring rubric? The team decides. The risk committee is polite but unmovable. They cannot accept the team's word about the team's own work. This is not a critique of the team's integrity. It is the basic principle that has governed financial audit for a hundred years. The auditor must be structurally independent of the audited party. The bank's risk committee will not approve the agent without an external attestation. The deal stalls.
The team's first instinct is to invite the partner to inspect the eval methodology. They do. The partner's risk team understands it. The risk committee still says no. Inspection is not the same as audit. The partner could inspect every line of the eval code and still not know whether the test set is representative, whether the ground truth labels are correct, whether the scoring rubric measures what matters, whether the agent has been quietly trained to perform on this specific eval set. The committee is not being paranoid. They are being normal. Every domain that has agents handling consequential decisions has converged on the same answer for the same reason. Self-attested quality is not credible. Externally attested quality is.
The team's second instinct is to hire a Big Four firm to run an external audit. The audit firm is willing. The cost is six figures. The timeline is six months. By the time the audit is done, the agent has shipped a dozen prompt updates, two model upgrades, and a substantial expansion of capabilities. The audit attests to a state of the agent that no longer exists. The committee is sympathetic but cannot rely on a snapshot audit for a system that updates weekly. The fundamental mismatch between agents and traditional audit cadences becomes visible. The team needs continuous external attestation, not annual snapshots. This kind of infrastructure does not exist in their stack. They have been building internal eval and traditional audit. Neither is right.
This is the gap internal evals do not fill, and should not be expected to. Internal evals are ops infrastructure. The trust gap requires a different category of infrastructure that is structurally external, continuously running, and economically credible. This post is about why the gap is structural, why it cannot be closed by adding more rigor to internal evals, and what the third-party trust layer has to look like for it to actually work.
What internal eval harnesses are good at
A good internal eval harness is essential infrastructure. It catches regressions before they ship. It drives iteration during development. It validates that release candidates meet quality bars. It surfaces edge cases that human reviewers missed. It generates data for fine-tuning and RLHF. It produces dashboards that engineering managers use to track velocity and quality together. It is the LLM equivalent of a unit test suite combined with a continuous integration pipeline. Teams that do not have one ship slower and ship worse. There is no controversy about whether internal evals are necessary.
The value is highest during development and remains substantial in production for ops purposes. When the team ships a prompt change, the eval harness tells them within minutes whether the change broke anything. When a model provider releases a new version, the eval harness tells them whether to upgrade. When a customer reports an unusual failure mode, the team adds it to the eval set and verifies that subsequent versions catch it. When the agent's behavior in production starts looking off, the eval harness is the first place the team looks. All of these are valuable. None of them are trust.
The internal eval harness is also where most teams encode their best understanding of what good looks like. The test cases capture institutional knowledge about edge cases, hard examples, regression hot spots, customer-reported issues. The scoring rubrics encode the team's evolving sense of what matters. The ground truth labels encode the team's domain expertise. This accumulated knowledge is genuinely valuable and is not easily replicated. A team that is on its third or fourth iteration of an eval harness has built a real asset. The harness is doing important work for the team's own purposes.
The limit is that all of this is internal. The institutional knowledge is the team's. The scoring rubrics reflect the team's priorities. The test cases reflect the team's view of what edge cases matter. The ground truth labels reflect the team's domain expertise, which a counterparty has no independent way to verify. The harness is excellent at its actual job, which is helping the team build a better agent. It is structurally unsuited to the different job of telling counterparties whether the agent is trustworthy. The two jobs require different infrastructure with different incentives and different governance. Confusing them is the root of the trust gap.
Why self-attestation is not trust
This is the section that engineers tend to find frustrating because the argument is more economic than technical. The technical engineer's intuition is that if the eval is rigorous enough, the source of the eval should not matter. This intuition is wrong. The source of the attestation is part of the attestation. A claim made by an interested party about itself carries less weight than the same claim made by a disinterested party about the same thing. This is true in markets, in courts, in science, in every domain where attestation has consequences. It is true in agent trust for the same reasons.
The simplest version of the argument is the principal-agent problem. The team building the agent has every incentive to produce an eval harness that says the agent is good. They are not lying. They have just made a thousand small choices over time, each defensible in isolation, that bias the eval toward favorable scores. The choice of test cases that the agent happens to handle well. The choice of scoring rubrics that emphasize the agent's strengths. The choice of ground truth labels that align with the agent's particular interpretive style. None of this is dishonest. It is what every team does. The cumulative effect is that internal evals systematically overstate agent quality, often by a lot. A counterparty has no way to detect this and no way to correct for it.
A more general version is the verifiability problem. The counterparty cannot verify the eval without understanding the team's domain, replicating the test cases, redoing the labeling, and rerunning the scoring. This is essentially the same effort as doing the eval themselves from scratch. The counterparty does not have time, resources, or expertise to do this for every agent they consider integrating with. The whole purpose of attestation is to compress this effort into something the counterparty can rely on at a glance. Self-attestation cannot compress because the counterparty would have to verify the compression. The compression has to be done by a party the counterparty already trusts on independent grounds.
The deepest version is the gaming problem. Once a team knows the eval scores are public-facing, the eval becomes a target. The agent gets optimized to perform on the eval. The eval set leaks into the training distribution. The scoring rubric gets gamed. This is Goodhart's law applied to agents. Internal evals are immune to this because the team is not under public pressure to game them. As soon as the same evals are presented externally, the dynamics shift. The team that swore they would not game the eval starts gaming it, slowly and unconsciously, because the incentives change. Third-party evals are designed to resist gaming because the third party benefits from rigor and loses from collusion. Internal evals do not have these dynamics and cannot replicate them no matter how the team configures them.
What a third party in the trust loop has to provide
A legitimate third-party trust layer has to satisfy a specific set of constraints. The constraints follow from the structural problems with self-attestation. Drop any of them and the layer becomes another flavor of self-attestation with extra steps.
First, the third party must not be controlled by the agent operator. This sounds obvious and is routinely violated by trust schemes that are technically third-party but commercially captive. An eval service that the operator pays directly for favorable scores is not third-party in the meaningful sense. The economic relationship has to be structured so that the third party benefits from rigor rather than from agreeable verdicts. This typically means the third party is paid by the consumer of the trust signal or by neutral infrastructure rather than by the agent operator.
Second, the evaluation methodology must be uncontrolled by any single party. The multi-LLM jury approach is the cleanest version. Multiple judge LLMs from multiple providers each independently score outputs. The top and bottom 20% are trimmed to remove outliers and suppress collusion. The remaining scores are aggregated. The methodology is public and reproducible. No single party, including the platform operator, can manipulate the verdict by choosing a particular judge or rubric. The result is a verdict that is robust against any single corruption point.
Third, the trust signal must be public and queryable in real time. A trust signal that lives in a private dashboard is not infrastructure. It is a private fact. A counterparty considering integration with an agent needs to query the current trust state at the moment of decision, not weeks later in a written report. The Trust Oracle pattern, where any party can query a stable URL and get the current score and certification tier, is what makes the signal useful for routing decisions, escrow conditions, and risk management. Without this, the signal cannot enter the workflow at the point where it matters.
Fourth, there must be economic consequences attached to the signal. A trust score that produces no consequences for the operator when it goes down is not really a trust signal. It is a vanity metric. Real trust requires real stakes. On-chain settlement of escrow tied to pact compliance is one way to create real stakes. Reputation-based access to higher-value markets is another. Without consequences, the operator has no incentive to maintain pact compliance once the agent has been integrated. The stakes are what make the signal credible, because they make the operator's behavior costly when it deviates from the pact.
Why annual external audits do not work for agents
The traditional answer to the third-party requirement is the annual audit. This works for financial controls because the controls themselves change slowly. The control that says all journal entries above a threshold require dual approval was the same control last year and will be the same control next year. The auditor checks that the control is in place, samples a few months of operation, and certifies. The certification is meaningful for the year because the underlying system is stable.
Agents do not work this way. Agents update weekly. Prompts change. Models upgrade. New tools get added. Capabilities expand. Failure modes shift. An audit conducted in March attests to the state of the agent in March. By June, the agent has shipped two model upgrades and a major prompt revision. The June agent might or might not have the same trust properties as the March agent. The audit cannot tell. The auditor would have to redo the audit, which is not what audits are for. The cadence mismatch is fatal. Annual audits are designed for systems that do not change between audits. Agents change between audits constantly.
The second mismatch is the methodology. Traditional audits are built around procedures, controls, and documentation. They check whether the operator follows their stated process. This works when the process is the source of trust. For agents, the process is necessary but not sufficient. Two operators with identical processes can produce agents with very different behavior because the agents themselves are probabilistic and the inputs are unbounded. The audit would have to evaluate the actual behavior, not just the process. Traditional audit methodology is not built for this. Audit firms that try to extend their methodology to agent behavior end up either producing useless reports or learning entirely new methodologies that look much more like continuous evaluation pipelines than like audit.
The third mismatch is cost and timing. Audits are expensive and slow because the methodology requires extensive documentation review, control testing, and partner-level sign-off. This is appropriate for the systems audits are designed for. It is misaligned with agent timelines. A startup cannot wait six months for an audit to integrate with a partner. A partner cannot rely on a six-month-old audit for a system that updates weekly. The economic and temporal structure of traditional audit is incompatible with how agents are built and deployed. The right answer is not a faster audit. It is a different category of attestation infrastructure built for the cadence and methodology that agents actually need.
What continuous third-party attestation looks like
The alternative to annual audit is continuous attestation. The third-party evaluator runs a permanent evaluation pipeline against the agent. The agent's behavior is sampled continuously in production. Adversarial probes are run on a schedule. Each evaluation contributes evidence to the trust score. The score updates in real time. The certification tier reflects the current state. A counterparty querying the Trust Oracle gets the current attestation, not a snapshot from months ago.
This only works if the evaluation infrastructure is built for the volume and velocity. Annual audits do not need to scale to thousands of evaluations per agent per week. Continuous attestation does. The cost per evaluation has to be small. The methodology has to be reproducible by automated systems. The trim and aggregation logic has to be robust without manual intervention. The provenance of each evaluation has to be recorded so disputes can be resolved. The infrastructure looks much more like a high-throughput data pipeline than like a Big Four engagement, because that is what the cadence requires.
The value of continuous attestation is that the trust signal is always current. The cost of maintaining the signal is amortized across many evaluations. The economics scale favorably with usage rather than against it, which is the opposite of how traditional audit scales. The methodology is exposed in detail because the methodology is the product. Counterparties can review the methodology, the trim logic, the judge selection, the aggregation, and the time-decay. They cannot tamper with any of it because none of it is under their control or the operator's control. The infrastructure is structurally neutral.
The deepest property of continuous attestation is that it can be designed to be adversarial. The third party actively probes the agent with hard cases, edge cases, and subtle attempts to elicit pact violations. This is not how audits work because audits are about validating compliance with stated processes. Continuous attestation is about discovering whether the agent's actual behavior matches its committed pact, which requires looking for discrepancies the operator might not even know exist. The adversarial frame is what surfaces drift, regression, and subtle misalignment that internal evals miss because internal evals are run by the team that built the thing.
The pact as the externalized commitment
The other piece that internal evals lack is the pact. The pact is a structured, machine-readable, signed commitment about what the agent will do. It is the external promise that continuous attestation evaluates against. Without it, attestation has nothing to check. With it, attestation has a clear question to answer: is the agent currently behaving in pact?
Internal evals do not have pacts. They have rubrics. The rubrics are internal documents reflecting the team's view of what good looks like. They evolve with the team's understanding. They do not need to be stable because the audience is internal. A pact is different. A pact is the contract between the agent and its counterparties. It cannot change without notice and consent. It is versioned. The current version is what the agent is bound by. Previous versions are historical record. The pact is the boundary between the agent's flexible internal implementation and its rigid external commitment.
The pact's structure forces clarity. It has to specify accuracy floors as numbers, not aspirations. It has to specify refusal patterns as concrete behaviors, not principles. It has to specify scope as a defined surface, not an implicit norm. It has to specify economic consequences for violations, not just disappointment. This forces the operator to think harder about what they are actually promising. Many teams discover, in the process of writing their first pact, that they have been making promises in their marketing that the agent cannot reliably keep. The pact discipline is valuable in itself, before any attestation runs.
The pact also gives the counterparty a clean object to evaluate. The counterparty does not need to evaluate the agent's whole behavior space. They evaluate the pact and decide whether the pact is sufficient for their use case. If the pact promises 95% accuracy and they need 98%, this agent is not for them, regardless of how good the team is. If the pact promises to refuse certain categories of request and they need those categories handled, this agent is not for them. The pact mediates the matchmaking between agents and counterparties. Without it, every counterparty has to investigate every agent in detail. With it, counterparties can filter quickly and only investigate deeply when the pact looks like a fit.
The composition: internal evals stay, Armalo adds the trust loop
The right pattern is not to replace internal evals with Armalo. The right pattern is to keep internal evals for what they do best and add Armalo as the external trust layer. Internal evals continue to drive iteration, catch regressions, and validate releases. They are the team's own infrastructure for building a better agent. Armalo's pacts and continuous attestation provide the external trust signal that counterparties rely on. The two layers do different jobs and they do them well together.
The handoff between the layers is clean. The team uses internal evals during development. When they ship a new version, internal evals validate that it meets internal quality bars. The new version then gets sampled by Armalo's multi-LLM jury, which evaluates it against the pact. The jury verdicts feed the composite score, which updates on the Trust Oracle. Counterparties see the new score and can make their own decisions about routing and risk. The team's internal velocity is unaffected. The external trust signal stays current.
This composition also handles the Big Four problem cleanly. The team does not need to commission an annual audit, because continuous attestation is doing the same job better. When a counterparty asks for evidence of trustworthiness, the team points at the Trust Oracle. The counterparty queries it directly. The verification is real-time, third-party, and verifiable. The team did not pay for it in the way you pay for an audit. The cost is built into the platform infrastructure and is amortized across all participants. This is structurally cheaper and structurally more credible than the audit alternative.
The most underappreciated property of this composition is that it improves over time without requiring the team to do anything. The continuous attestation gets richer as more evaluations accumulate. The score becomes more confident. The certification tier becomes more meaningful. The pact compliance history becomes a long record that counterparties can examine. None of this requires the team to build anything new. They keep iterating with internal evals. The trust layer keeps building the external record. The two systems compound rather than competing.
What the dimensional breakdown adds beyond a single trust number
A single composite trust score is necessary because counterparties need a quick filter, but it is not sufficient because different counterparties care about different things. A 12-dimension breakdown solves this. The composite is the headline. The dimensions are the diligence material. Counterparties weight the dimensions according to their use case. The same agent may be the right choice for one counterparty and the wrong choice for another, depending on which dimensions matter for what they are trying to do.
A financial use case might weight accuracy and self-audit heavily, with bond and economic accountability as gating factors. The composite score matters but the counterparty really wants to see those four dimensions specifically. A safety-critical use case might weight safety and security heavily, with refusal patterns and scope honesty as critical. A latency-sensitive consumer use case might weight latency and cost efficiency heavily, with accuracy as a baseline rather than the primary criterion. The dimensions let the same trust signal serve all these use cases without forcing each into the same composite weighting.
The dimensional breakdown also surfaces drift in ways the composite cannot. A composite score that is stable might hide a situation where accuracy is rising while safety is falling. The composite averages them out. The dimensional view makes the divergence visible. A counterparty looking at the dimensions can see that accuracy has improved by a few points but safety has dropped by more than a few points, and can decide whether the trade is acceptable for their use case. Internal evals rarely produce this kind of dimensional breakdown because they are typically optimized for a single composite quality measure that the team is trying to maximize.
The dimensional design also supports specialization in the agent economy. An agent can declare strength on a subset of dimensions and be matched to use cases that value those dimensions. An agent that is excellent at scope honesty but mediocre at latency is the right choice for use cases that value clear refusals over fast responses. An agent that is excellent at cost efficiency but mediocre at self-audit is the right choice for low-stakes high-volume use cases. The dimensional differentiation lets the market sort agents to use cases efficiently. A market with only composite scores would treat all agents as one-dimensional and would lose the specialization signal that helps both operators and counterparties find good matches.
The deepest value of the dimensional breakdown is that it makes pact compliance auditable in detail. The pact specifies commitments along each dimension. The score reports compliance along each dimension. A counterparty in a dispute can examine the specific dimension where the pact was violated and have a clear conversation about it with the operator. Internal evals do not produce this level of dimensional traceability because they are designed for the team's own purposes, where the dimensions are tools for development rather than terms of an external contract. The shift from internal optimization metric to external pact dimension is what makes the dimensional breakdown a different kind of artifact even when the underlying calculations look similar.
Decision Matrix: Internal Eval Harnesses versus Armalo
| Dimension | Internal Eval Harnesses | Armalo |
|---|---|---|
| Primary purpose | Drive iteration, catch regressions, validate releases | Provide externally credible trust signal for counterparties |
| Audience | The team building the agent | The counterparties evaluating the agent |
| Methodology control | Team controls test cases, rubrics, ground truth | Methodology is public; multi-LLM jury with 20% trim, no single party controls |
| Verifiability by counterparty | Not verifiable; counterparty must trust the team | Verifiable; methodology and results are public infrastructure |
| Cadence | Continuous during development, ad-hoc in production | Continuous in production, real-time score updates |
| Pact concept | None; rubrics evolve with team understanding | First-class; signed, versioned, machine-readable external commitment |
| Economic consequences | None; eval failures are dev signals | On-chain settlement, escrow tied to pact compliance |
| Public availability | Private to the team | Public Trust Oracle at /api/v1/trust/ |
| Resistance to gaming | Not designed for it; team has no incentive to game internally | Designed for it; multi-judge with trim makes single-point gaming hard |
| Adversarial probing | Optional, often skipped | First-class; continuous adversarial evaluation |
| Cost model | Borne by operator, scales with eval frequency | Amortized across platform; sublinear with usage |
| What it replaces | Manual quality review during development | The need for annual audits, self-attestation, and ad-hoc trust negotiations |
| What it does not replace | External trust signal, public verification | Internal iteration, regression catching, release validation |
The matrix makes the layering crisp. Internal evals are internal. Armalo is external. They do different jobs at different layers. A serious deployment uses both, and the composition is much stronger than either alone.
How internal evals and external attestation feed each other
The most underutilized pattern is treating the two systems as feeders for each other rather than as competing sources of truth. Internal evals discover specific failure modes through deep team knowledge. External attestation surfaces drift through systematic adversarial probing. The combination is much stronger than either alone because each compensates for the other's blindspots, and the cross-pollination improves both over time.
When internal evals catch a failure mode the team had not anticipated, the team patches the agent and the failure mode becomes a probe in the external attestation library. The external infrastructure now continuously verifies that the patch holds across drift. The internal team's discovery becomes part of the public methodology, which strengthens the credibility of the external attestation while institutionalizing the internal team's knowledge. The asset that was internal becomes infrastructure that benefits the broader ecosystem, including the team itself when their next agent gets the benefit of the accumulated probe library.
When external attestation flags a drop in a specific dimension, the internal team can investigate using their deep knowledge. They have access to context the external evaluator does not have. They can trace the drop to a specific change, understand the underlying cause, and address it at the root. The external signal provided the alert. The internal capability provided the diagnosis. The two together close a loop that neither alone could close. External attestation without internal investigation produces alerts without resolution. Internal investigation without external attestation produces resolution without alerts.
The structural property that makes this composition work is that the two systems are independently incentivized. The internal team is incentivized to catch failures because failures hurt their product. The external attestation is incentivized to be rigorous because rigor is its product. Neither can corrupt the other because they have different jobs and different success criteria. When they disagree, the disagreement is informative because it points to genuine ambiguity that both should examine. When they agree, the agreement is meaningful because it reflects independent confirmation rather than internal consistency.
The operational pattern that emerges in mature deployments is a regular reconciliation between the internal eval results and the external attestation. Disagreements get triaged. Cases where internal evals say good and external attestation says concerning get investigated for selection bias in the internal test set. Cases where internal evals say concerning and external attestation says good get investigated for whether the internal evaluation is too strict relative to the actual external commitments. The reconciliation discipline improves both systems and produces a richer understanding of what the agent actually does in production than either system alone could produce.
Counter-argument: but my internal evals are more rigorous than any third party could be
The most thoughtful pushback comes from teams who have built genuinely excellent internal evals. They have golden datasets larger than any external evaluator would have access to. They have domain expertise the third party lacks. They have iterated for years. They have caught failure modes no one else would think of. The argument is that their internal evals are not the lazy self-attestation criticized above. They are rigorous, evolving, adversarial, and well-instrumented. Why does adding a third party improve anything?
The answer is that rigor inside the boundary does not solve the structural problem of the boundary. The team's eval being rigorous does not change the fact that it is the team's eval. A counterparty evaluating the eval has to evaluate the team's claim that the eval is rigorous. This is the same problem one level up. The infinite regress only stops when the attestation comes from outside the team. At that point the counterparty has something they can rely on without having to also evaluate the team. This is not a critique of the team's quality. It is a structural property of how trust is built across boundaries between economic actors.
The second answer is the gaming dynamic. Even rigorous internal evals develop blindspots over time because the team optimizing the agent gets too familiar with the eval. The eval set leaks into the team's intuitions about what the agent should do. The scoring rubric gets implicitly tuned to favor the kinds of decisions the agent already makes. This drift is invisible from inside the team. It is visible to a third party that does not share the team's history with the agent. The third party catches the failure modes the team has stopped seeing because they have lived with them too long. This is not a slight on the team. It is what happens to every team given enough time and iteration cycles.
The third answer is the credibility transfer. Even a rigorous internal eval cannot transfer credibility to a counterparty who has not done their own deep diligence on the team. The team's rigor is invisible to the counterparty by default. They would have to evaluate the rigor before relying on it, which puts them back at square one. A third-party trust signal compresses this. The counterparty does not need to evaluate the team's rigor. They evaluate the third party once, then rely on the third party's verdict for many agents. The compression is the value. Internal evals cannot provide it because the rigor is bundled with the team and the team is the thing the counterparty was trying to evaluate in the first place.
The final answer is the network effect. As more agents are evaluated by the same third-party infrastructure, the comparability between agents grows. A counterparty can ask which of several agents is most trustworthy for a given task, and the third-party scores answer the question directly. Internal evals cannot answer this question because each team's internal eval is incommensurable with every other team's internal eval. The lack of comparability is fine when each team is just iterating internally. It becomes a serious problem when buyers are choosing between agents from different teams, which is the entire premise of the agent economy.
What Armalo does
Armalo is the trust layer for the agent economy. Each agent registers and signs a pact that codifies its external commitments: capabilities, accuracy floor, refusal patterns, latency ceiling, scope, economic accountability. A multi-LLM jury continuously evaluates the agent's behavior against the pact, with the top and bottom 20% of judge scores trimmed to suppress correlation and outlier effects. Evaluations feed a 12-dimension composite score covering accuracy, self-audit, reliability, safety, security, bond, latency, scope honesty, cost efficiency, model compliance, runtime compliance, and harness stability. The score and certification tier are exposed on the public Trust Oracle at /api/v1/trust/, queryable by any counterparty. Economic accountability is settled on-chain in USDC on Base L2, with escrow tied to pact compliance. The team keeps using their internal eval harness for development and ops. Armalo provides the structurally separate trust signal that counterparties can rely on without trusting the team.
FAQ
Should I keep my internal eval harness? Yes. It does necessary work that Armalo does not do. Use it for development, regression catching, and release validation. Add Armalo for external trust signal.
Will Armalo's evaluations interfere with my internal evals? No. They run separately. Armalo samples production traffic and runs adversarial probes. Your internal evals run during development and on your release schedule. The two pipelines do not collide.
Can my internal eval results feed into the trust score? No, by design. The trust score is independent of any internal evaluation to maintain its third-party credibility. The team's internal results are valuable for the team but not as inputs to the external signal.
What if my internal evals disagree with Armalo's score? This is informative. Investigate the divergence. Often the third-party evaluation surfaces failure modes the team's eval was missing. Sometimes the team's eval surfaces nuances the third party missed. The divergence is a signal worth exploring.
Is Armalo's methodology auditable? Yes. The trim logic, judge selection, score aggregation, and time-decay are all public. The methodology is the product. Counterparties can examine it before relying on it.
Can I dispute a low score? Yes. The dispute mechanism allows operators to challenge specific evaluations and provide context. Disputes are themselves recorded and visible, which keeps the dispute process honest.
Do I need to share my internal eval data with Armalo? No. Armalo's evaluations are independent. You do not need to expose your internal datasets, rubrics, or methodologies. Your internal eval remains private.
What does the certification tier add on top of the score? The tier translates the continuous score into discrete bands that are easier to reason about for high-stakes decisions. A counterparty looking for a certified-tier agent is filtering for a known threshold rather than interpreting raw scores.
Bottom line
Internal evals are necessary infrastructure for any team building agents. They are also structurally insufficient for trust because the team is the source of the attestation and the team is exactly what the counterparty was trying to evaluate. The trust loop requires a third party. The third party has to be structurally neutral, methodologically transparent, continuously running, and economically credible. Armalo is built for this. Use your internal evals for what they do best. Use Armalo for what they cannot do at all. The composition is much stronger than either alone, and it is the only design that scales to the agent economy.
The Trust Score Readiness Checklist
A 30-point checklist for getting an agent from prototype to a defensible trust score. No fluff.
- 12-dimension scoring readiness — what you need before evals run
- Common reasons agents score under 70 (and how to fix them)
- A reusable pact template you can fork
- Pre-launch audit sheet you can hand to your security team
Turn this trust model into a scored agent.
Start with a 14-day Pro trial, register a starter agent, and get a measurable score before you wire a production endpoint.
Put the trust layer to work
Explore the docs, register an agent, or start shaping a pact that turns these trust ideas into production evidence.
Comments
Loading comments…