The A2A Disagreement Problem: When Two Agents Reach Different Conclusions From Same Inputs

TL;DR

Agent A says yes. Agent B says no. They received the same inputs and they returned different answers. The orchestrator now has to make a decision, and most A2A architectures have no defined protocol for what that decision should be. The default is for whichever agent the orchestrator queried first to win, which is operationally arbitrary and produces outcomes that depend on routing order rather than correctness. The right protocol has three steps: run a tie-break jury that synthesizes a third independent answer with explicit reasoning, escalate to human review when the jury cannot resolve, and abort the entire interaction when even human review cannot establish ground truth. Each step has specific design constraints that determine whether the protocol works or whether it just adds latency and theater. This post specifies the disagreement protocol, walks through the four common disagreement patterns and which protocol step handles each, and provides the decision tree that any production multi-agent orchestrator should implement.

Intro: The Two Agents Who Could Not Both Be Right

The compliance check was supposed to be simple. The orchestrator was processing a routine vendor onboarding request and needed to verify that the vendor's documentation met the regulatory requirements for the jurisdiction. The orchestrator routed the verification sub-task in parallel to two specialist compliance agents that had been added to the registry several months earlier, both with strong reputations and reasonable scores. The standard practice was to require both agents to confirm before allowing the onboarding to proceed.

Agent A returned a clear pass. The documentation was complete, the entity was registered correctly, the disclosures were in order. Agent B returned a clear fail. The documentation was incomplete, a required disclosure was missing, the entity registration appeared to be lapsed. Both agents had received exactly the same documentation. Both were operating on pacts that authorized the same capability. Both responses were structurally well-formed, internally consistent, and signed by the agents' DIDs. Neither response gave the orchestrator any obvious reason to prefer it over the other.

The orchestrator's protocol had no defined behavior for this case. The implementing team had assumed that two specialist agents with strong reputations would naturally agree on routine verifications, and that disagreement was a rare edge case that could be handled manually. In practice, disagreement turned out to be common enough that manual handling was not viable: roughly two percent of compliance checks produced disagreements, which at the orchestrator's volume meant several disagreement events per day. The team's stopgap was to default to whichever agent had responded first, which was almost always agent A because of its slightly faster average latency. This default meant agent B's perspective was systematically discounted, even though there was no evidence that agent A was more reliable than agent B on the specific question at issue.

When the team eventually audited the disagreement cases, they found that both agents were sometimes correct and both agents were sometimes wrong. Defaulting to agent A had produced worse outcomes than coin-flipping would have, because agent A had a particular pattern of false negatives that the team had not detected. The arbitrary tie-break had created a systematic bias that hurt the orchestrator's overall accuracy. By the time the team noticed, several months of decisions had been influenced by the bias, and unwinding them was expensive.

This is the disagreement problem in its purest form. When two agents that should be operating from the same information reach different conclusions, the orchestrator faces a structural decision that the protocol does not define. The default behaviors that orchestrators fall back on are usually arbitrary and often biased. The right answer is to have a defined protocol that handles disagreement explicitly, with mechanisms that produce defensible outcomes rather than arbitrary ones. This post specifies what that protocol looks like and why each component is necessary.

Why Disagreement Is Common, Not Rare

The first thing to understand about disagreement is that it is not a sign of agent malfunction. Two well-functioning LLM-based agents that are operating on the same inputs will often reach different conclusions, especially on judgment-laden questions, even when both are operating correctly within their pacts. The disagreement is a feature of the agents' reasoning processes, not a bug in any one of them. Architectures that treat disagreement as exceptional are architectures that will be surprised by how often it happens.

The sources of disagreement are several. LLMs are non-deterministic by default, and even with temperature zero settings the same model can produce different reasoning chains on different invocations because of the underlying numerical sensitivity in attention computation. Different models, even when fine-tuned for the same task, embed different priors about what counts as evidence and what counts as conclusion. Different system prompts, even when they appear to specify the same task, can shift the model's emphasis in ways that change the answer on edge cases. Different tool surfaces give different agents access to different information, even when the question they are answering nominally only requires the same input documents.

The ambiguity of the underlying questions is a deeper source of disagreement. Many of the questions that agents are asked to answer are not questions with a single correct answer. "Is this documentation compliant?" is a question whose answer depends on which interpretation of the regulatory framework the answerer applies. "Is this code review thorough enough?" depends on what counts as thoroughness. "Is this market analysis accurate?" depends on the time horizon and the comparison set. Two agents that disagree on these questions are not necessarily wrong. They may be applying different reasonable interpretations to a question that admits multiple reasonable answers.

The practical implication is that the orchestrator needs to expect disagreement on a meaningful fraction of interactions, and to have a protocol that handles it without falling back on arbitrary tie-breaking. The fraction will be higher for judgment-laden capabilities and lower for fact-checking capabilities, but it will not be zero in any plausible production setting. Architectures that treat disagreement as an edge case are architectures that have not yet looked at their own data carefully.

The deepest source of disagreement is the operator-side variation that the orchestrator has limited visibility into. An agent's behavior is a function of its model, its system prompt, its tools, and the operator's ongoing tuning. Two agents that look interchangeable on paper can have meaningfully different behaviors because of accumulated tuning that the registry does not surface. This variation is not a problem to eliminate. It is the reason for using multiple agents in the first place: different agents catch different things, and the disagreement protocol is what extracts value from the diversity rather than treating it as noise.

The First Step: The Tie-Break Jury

The first step in any disagreement protocol is to introduce additional independent perspectives. Two agents disagreeing is a binary stalemate. Adding a third independent answer breaks the stalemate, as long as the third answer is genuinely independent and not just a tiebreaker that defaults to one side. The mechanism for producing the third answer is a multi-LLM jury, structured to be both independent and explicit about its reasoning.

The composition of the jury matters. A jury that consists of agents from the same operator as either agent A or agent B is not independent. A jury that consists of agents using the same underlying model as either agent A or agent B is partially independent at best, because shared model priors will pull the jury toward shared conclusions. The robust jury is composed of agents with diverse model bases, diverse operators, and diverse capability heritage. The heterogeneity is what gives the jury its independence.

The size of the jury matters. A jury of three is the minimum for breaking ties but is sensitive to outliers. A jury of seven is the typical sweet spot: large enough to absorb individual outlier judgments, small enough to keep latency and cost manageable. A jury of fifteen or more is reserved for high-stakes disputes where the cost of a wrong outcome justifies the additional inference. The orchestrator's protocol should specify the jury size as a function of the value at stake, with larger juries triggered automatically for higher-stakes disputes.

The aggregation rule matters. Simple majority voting is not robust because it is sensitive to the framing of the question. The robust aggregation is to trim the top and bottom 20% of judgments and average the middle, which discards extreme outliers and prompt-injection attempts while preserving the central tendency of the jury. This is the same trimming pattern that Armalo's composite scoring uses for similar reasons. The trimming reduces the jury's susceptibility to manipulation by any single outlier judgment.

The explicit reasoning requirement matters. A jury that returns only a yes or no answer provides less value than a jury that returns a yes or no plus the reasoning behind it. The reasoning is what lets the orchestrator (or a downstream human reviewer) understand why the jury concluded what it concluded, and whether the jury's reasoning addresses the specific points where agents A and B disagreed. A jury that ignores the points of disagreement is a jury that has not actually adjudicated the disagreement, even if it has produced an answer.

The deepest aspect of the tie-break jury is the question of what to do when the jury itself is internally divided. A jury that splits 4-3 on a difficult question is not the same as a jury that comes back unanimous. The protocol should distinguish these cases. Unanimous or near-unanimous juries can be treated as resolving the disagreement. Split juries should be treated as a sign that the question itself is harder than the original two-agent disagreement suggested, and the protocol should escalate rather than declaring the slim majority the winner.

The Second Step: Human Escalation

When the tie-break jury cannot produce a confident resolution, the disagreement escalates to human review. This step is not a fallback because LLM juries are unreliable. It is a fallback because some disagreements involve questions that LLMs are not currently well-equipped to resolve, and forcing the LLM jury to produce an answer in those cases is asking for an arbitrary outcome dressed up as a reasoned one.

The trigger for human escalation should be specific. A jury that splits more than a defined threshold (typically 60-40 or closer) is one trigger. A jury that produces low-confidence reasoning (acknowledging uncertainty in the reasoning chain itself) is another. A disagreement that involves a value at stake above a defined threshold, regardless of jury confidence, is a third. A disagreement that has been escalated to human review previously and produced a reversal of the LLM consensus is a fourth, because it indicates the topic is one where LLM judgment is systematically off.

The human reviewer's role is not to make the decision in isolation. The reviewer should receive the original inputs, the original disagreement, the jury's reasoning, and any relevant context from the trust oracle about the agents involved. The reviewer's job is to make a defensible decision that is informed by all of this, not to start from scratch. Reviewers who start from scratch are reviewers whose decisions cannot be efficiently scaled, because they reproduce work that the LLM layer has already done.

The reviewer's decision becomes a signal that feeds back into the system. The agents whose answers were correct accrue reputation. The agents whose answers were incorrect lose it. The jury whose reasoning aligned with the human's reasoning is rewarded. The jury whose reasoning diverged is reweighted. The pact whose acceptance criteria were too vague to support a clear answer is flagged for revision. The whole disagreement event becomes data that improves future disagreements, which is what closes the recursive improvement loop.

The deepest aspect of human escalation is the question of who the reviewer is. An ad-hoc reviewer pulled from the orchestrator's operations team will produce inconsistent decisions. A trained reviewer who specializes in the capability area will produce more consistent ones. A panel of reviewers with rotating membership will produce decisions that resist individual reviewer bias. The right structure depends on the volume of escalations and the stakes involved, but the principle is that human review should be a structured process with explicit standards, not a casual eyeball check by whoever is on call.

The Third Step: Strategic Abort

The third step is the one that orchestrator teams are most reluctant to add because it requires admitting that some interactions cannot be completed. When the human reviewer cannot establish ground truth, when the question at issue is genuinely ambiguous, or when the cost of getting the answer wrong exceeds the cost of not answering at all, the protocol should abort the interaction. The orchestrator returns to the requester with an explicit statement that the question cannot be answered to the orchestrator's satisfaction.

This step is structurally important even though it feels like a failure. The alternative is forcing a decision in cases where no defensible decision is available, which produces worse outcomes than the abort would have. An orchestrator that always returns an answer is an orchestrator whose answers are sometimes invented, and the requester has no way to distinguish the cases where the orchestrator is confident from the cases where it is bluffing. An orchestrator that aborts when appropriate gives the requester a meaningful signal about the orchestrator's actual epistemic state.

The abort should be specific about why it is happening. "The compliance check could not be completed because two specialist agents disagreed and the tie-break jury split 4-3 with low confidence reasoning, and human review concluded that the documentation is genuinely ambiguous on the requirement in question" is a useful abort. "Sorry, we could not process your request" is not. The specificity is what lets the requester decide whether to gather additional information, escalate to a different verification path, or proceed at their own risk with the ambiguity acknowledged.

The abort should also include any partial information that the orchestrator did establish. If both agents agreed on most of the documentation and disagreed only on one specific requirement, the abort should note that. The requester can then focus their attention on the specific point of disagreement rather than re-running the entire verification. Partial information is not nothing; it is the most useful signal that an aborting orchestrator can provide.

The deepest aspect of the abort decision is the question of what counts as a high enough cost of getting the answer wrong to justify aborting. This is a parameter the orchestrator's risk policy has to set. Compliance verifications where a wrong answer produces regulatory consequences should have low abort thresholds. Recommendation systems where a wrong answer produces a slightly suboptimal user experience can have high abort thresholds. The threshold is not a universal value; it is a per-capability decision that the orchestrator should make explicitly and document, because the alternative is making the decision implicitly and inconsistently.

The Four Common Disagreement Patterns

Disagreements have characteristic shapes. Recognizing the shape of a disagreement makes the protocol more efficient because each shape benefits from a slightly different handling. Four shapes are common enough to be worth naming explicitly.

The interpretation disagreement is the case where both agents see the same facts but apply different interpretive frameworks. Agent A reads the regulatory requirement strictly and concludes the documentation fails. Agent B reads the requirement permissively and concludes it passes. The disagreement is not about facts; it is about which interpretation to apply. The tie-break jury can resolve this if the jury is given explicit guidance on which interpretive framework to apply. Without that guidance, the jury will reproduce the disagreement at the next level.

The evidence disagreement is the case where the agents have different views of what the relevant facts are. Agent A bases its conclusion on the documentation provided. Agent B has access to an external database that provides additional evidence not present in the documentation. The disagreement is about what counts as relevant input. The tie-break jury cannot resolve this on its own; the orchestrator first has to reconcile the input sets, either by giving both agents access to the same evidence or by explicitly choosing which evidence set governs.

The quality disagreement is the case where one agent is doing better work than the other on this specific question. Agent A is using a more capable model, has been better tuned for this capability, or has access to better tools. Agent B is producing a less rigorous answer because of capability gaps. The disagreement reflects a real difference in agent quality, not just a difference in interpretation. The tie-break jury will typically side with the higher-quality agent, but the orchestrator should also use the disagreement as a signal to reconsider whether agent B should still be on the roster for this capability.

The value-judgment disagreement is the case where the question itself involves a value judgment that has no objectively correct answer. "Is this risk acceptable?" is a question whose answer depends on the requester's risk tolerance, which the agents may have different priors about. The tie-break jury cannot resolve this; the orchestrator has to escalate to the requester or to a human reviewer who can elicit the requester's actual values. Forcing an answer in this case is producing an arbitrary outcome.

The deepest aspect of the patterns is that real disagreements often combine multiple shapes. An interpretation disagreement that also involves a quality difference is harder to resolve than either alone, because the protocol has to address both dimensions. The orchestrator's classification of disagreements into shapes should be explicit (logged with each disagreement event), so that the resolution path can be matched to the shape, and so that aggregate analysis can reveal which shapes are most common in which capability areas.

Reader Artifact: The A2A Disagreement Resolution Decision Tree

When two or more agents return different conclusions from the same inputs, follow this decision tree. Each branch leads to a specific action. The tree is intentionally explicit because implicit decision-making in disagreement cases is what produces biased and indefensible outcomes.

Node one: confirm the disagreement is genuine. Were the inputs identical? Were the pacts compatible? Are the responses semantically different (not just stylistically different)? If any answer is no, reconcile the inputs or align the pacts before treating the responses as a real disagreement. Pseudo-disagreements that are actually input mismatches do not need disagreement resolution.

Node two: classify the disagreement shape. Is this an interpretation disagreement, an evidence disagreement, a quality disagreement, or a value-judgment disagreement? The classification determines which downstream resolution path is most appropriate. If the shape is unclear, default to the most rigorous path (full tie-break jury with explicit reasoning) and refine the classification once the jury reasoning provides more signal.

Node three: evaluate the value at stake. Is the cost of getting this wrong above the threshold for full resolution? If yes, proceed with the full protocol regardless of the shape. If no, run a lightweight version: smaller jury, faster resolution, lower cost. The orchestrator's risk policy defines the threshold.

Node four: run the tie-break jury. Convene a jury of at least seven LLMs from diverse model bases and operators, give them the same inputs and an explicit prompt to address the points of disagreement, and trim the top and bottom 20% of judgments. Aggregate the middle. Capture each jury member's reasoning, not just the verdict.

Node five: evaluate jury confidence. Was the jury near-unanimous? Did the surviving judgments agree on reasoning, not just verdict? If yes, the jury has resolved the disagreement and the orchestrator can proceed with the jury's conclusion. If no, escalate to node six.

Node six: escalate to human review. Package the original inputs, the original disagreement, the jury's reasoning, and any relevant agent reputation data. Route to a trained reviewer. The reviewer's job is to make a defensible decision informed by all of this, with explicit reasoning that is documented for future reference.

Node seven: evaluate reviewer confidence. Did the reviewer reach a defensible conclusion? If yes, proceed with the reviewer's conclusion and record the outcome as a signal that updates the agents' reputations and the jury's calibration. If the reviewer concluded that the question is genuinely undecidable, escalate to node eight.

Node eight: strategic abort. Return to the requester with an explicit statement of why the interaction could not be completed, the partial information that was established, and the specific point of disagreement. Suggest paths the requester can pursue to get a defensible answer (additional verification, different evidence sources, accepting the ambiguity). Do not invent an answer to avoid the abort.

Node nine: post-resolution feedback. Regardless of which node terminated the tree, capture the outcome as a signal. Update agent reputation based on which agents' conclusions matched the eventual resolution. Update jury calibration based on whether the jury's reasoning matched the human reviewer's reasoning. Flag pacts whose acceptance criteria contributed to the disagreement for revision. Update the orchestrator's classification model based on how the disagreement shape was resolved.

This tree is the canonical decision logic for handling disagreement in any production multi-agent orchestrator. It is intentionally specific, intentionally branched, and intentionally documented. Implicit alternatives produce biased outcomes. Explicit alternatives produce defensible ones, even when the defensible outcome is to abort.

Counter-Argument: Why Some Argue The Disagreement Protocol Is Overkill

The strongest version of the counter-argument is that the disagreement protocol adds latency and complexity that most A2A interactions do not need, and that simpler approaches (single-agent routing, first-response wins, trust the higher-scoring agent) work well enough in practice. The argument continues that the cost of running the full protocol on every interaction would dwarf the cost of the occasional wrong answer that simpler approaches produce.

The response is that the protocol is not meant to run on every interaction. It is meant to run when disagreement actually occurs, which is a fraction of interactions in most use cases. For interactions where only a single agent is consulted, there is no disagreement and no protocol overhead. For interactions where multiple agents agree, the protocol terminates immediately at node one with no further work. The protocol's full machinery only engages when there is actual disagreement to resolve, which is exactly when the cost is justified.

The weaker version of the counter-argument is that the protocol's reliance on multi-LLM juries and human review introduces dependencies that make A2A architectures more complex than they need to be. The response is that the dependencies exist whether the protocol acknowledges them or not. Orchestrators that do not have an explicit jury and human review path still face disagreements, and they handle them through ad-hoc mechanisms that are usually less reliable than the explicit protocol would be. The complexity is not added by the protocol; it is exposed by the protocol. Hidden complexity is more dangerous than visible complexity, because hidden complexity cannot be reasoned about systematically.

The deepest version of the counter-argument is one that questions whether multi-agent disagreement is the right architecture in the first place. The argument is that orchestrators should rely on single agents whose pacts and reputations make them reliable enough that disagreement-free single-agent answers are good enough. The response is that single-agent architectures concentrate risk on the chosen agent's failure modes, which can be worse than the alternative. Multi-agent architectures with disagreement resolution are more robust because they catch failure modes that any individual agent might miss. The right architecture depends on the use case, but for high-stakes interactions, the multi-agent pattern is usually safer despite its added complexity.

What Armalo Does Here

Armalo provides the infrastructure that makes the disagreement protocol practical. The multi-LLM jury system is built into the platform, with configurable jury composition, automatic top-and-bottom 20% trimming, and reasoning capture for every jury member. Orchestrators can invoke the jury at sub-second to several-second latencies depending on jury size, with cost scaling linearly with the number of jury members.

The trust oracle at /api/v1/trust/ provides the agent reputation data that informs human review when escalation is required. Reviewers can see each disagreeing agent's score history, recent behavioral baseline, and historical performance on similar disagreement events. This context is what lets reviewers make informed decisions efficiently rather than starting from scratch on every escalation.

The deeper integration is the feedback loop that closes the disagreement protocol. Disagreement events are logged with their resolution paths, which feeds back into agent reputation calculation, jury calibration, and pact quality assessment. Pacts whose acceptance criteria contribute to recurring disagreements are flagged for revision. Agents whose conclusions consistently align with eventual resolutions accrue reputation. The whole system improves over time because every disagreement event is an input to the next round of reputation and calibration updates. This recursive improvement is what makes the protocol's cost worth paying. Without the feedback loop, disagreements would be one-off resolutions. With it, they become signal that improves the entire network's reliability.

FAQ

When should an orchestrator use multiple agents instead of a single agent? When the cost of a wrong answer is high enough to justify the additional inference cost, when the underlying capability is judgment-laden enough that diverse perspectives add value, or when the orchestrator wants defense in depth against single-agent failure modes. Single-agent architectures are appropriate for low-stakes, fact-based capabilities where reliable single agents exist.

How large should the tie-break jury be? Seven is the typical sweet spot. Three is the minimum for breaking ties. Fifteen or more is reserved for high-stakes disputes. The size should be a function of the value at stake, set by the orchestrator's risk policy.

What does "top and bottom 20% trimming" actually mean for a jury of seven? It means the highest one judgment and the lowest one judgment are discarded, and the middle five are aggregated. For a jury of fifteen, the top three and bottom three are trimmed and the middle nine are aggregated. The trimming proportion is consistent; the absolute count scales with jury size.

What happens if the jury is split 4-3 on a difficult question? The protocol treats this as low confidence and escalates to human review. A close split is a signal that the question itself is harder than the original disagreement suggested, and forcing the slim majority to win produces arbitrary outcomes.

How do you prevent jury members from being correlated through shared model bases? The orchestrator should require diverse model bases when convening the jury. A jury composed entirely of agents using the same underlying model is not independent and will produce correlated judgments. The trust oracle exposes model identifier information that lets the orchestrator enforce diversity at jury composition time.

What about juries themselves disagreeing? They should. If a jury always agreed, it would not be providing diverse perspectives. Disagreement within the jury is data that informs the aggregation. The aggregation rule (trim and average) is designed to handle disagreement gracefully. Unanimous juries are nice when they happen but should not be the expected outcome on hard questions.

Can the protocol handle disagreements between more than two agents? Yes. The classification, jury, and resolution steps are the same regardless of how many initial agents disagree. The jury's job is to produce an independent third (or fourth, or fifth) perspective, and the aggregation handles whatever number of original agents were involved.

What if the human reviewer themselves is wrong? The reviewer's decision is treated as ground truth for the immediate interaction, but it is also captured for downstream review. If aggregate analysis later reveals that a particular reviewer's decisions are systematically incorrect, the orchestrator can re-weight that reviewer's input and revisit the affected interactions. Human review is not infallible. It is the last layer of defense, with its own accountability mechanisms.

Bottom Line

Disagreement between agents is common, not rare. The orchestrator that does not have an explicit protocol for handling it falls back on arbitrary defaults that produce biased outcomes. The right protocol has three steps: tie-break jury with diverse composition and explicit reasoning, human escalation when the jury cannot resolve confidently, strategic abort when even human review cannot establish ground truth. The decision tree above specifies the branches and the actions at each node. Armalo provides the multi-LLM jury infrastructure, the trust oracle that informs human review, and the feedback loop that turns disagreement events into signal for ongoing reputation and calibration updates. Implement the protocol explicitly. Document the abort threshold for each capability. Capture every disagreement as data. The protocol's value compounds over time because every disagreement makes the next one easier to resolve. Networks that resist this protocol because it adds complexity end up with all the complexity hidden in ad-hoc handling that is harder to reason about. Networks that embrace it end up with multi-agent architectures that are actually safer than single-agent ones, which is the entire point of using multiple agents in the first place.