The Procurement Conversation: Forty Questions Every Buyer Should Ask About An Agent
Most agent procurement processes ask the wrong questions. Here are the forty that actually predict whether the agent will work in your environment, grouped by what they're testing.
Continue the reading path
Topic hub
Runtime GovernanceThis page is routed through Armalo's metadata-defined runtime governance hub rather than a loose category bucket.
TL;DR
Most agent procurement conversations get hung up on the wrong dimensions. Buyers ask about model architecture, leaderboard scores, and demo capabilities. Operators field those questions and everyone leaves the call with the wrong picture of how the agent will perform under load. The forty questions in this essay are the ones that actually correlate with whether the agent will work in the buyer's environment over a year of operation. They are grouped by what they test: identity (is the agent who you think it is), capability (can it do the job), trust (will it tell you when it can't), bond (does it have skin in the game), dispute (what happens when it's wrong), recovery (what happens when it breaks), and operational (can your team work with it). A buyer who asks all forty before signing will avoid most of the bad outcomes that incident-driven procurement learning produces.
Why the standard procurement conversation fails
The typical agent procurement conversation has three phases. First, the demo: the operator shows the agent doing something impressive. Second, the spec sheet: the buyer reads through the agent's claimed capabilities and ticks off the requirements. Third, the contract: legal review focuses on standard SaaS terms — uptime SLAs, data processing addenda, indemnification.
None of these phases test the things that actually matter for agent performance. The demo shows what the agent can do on a curated example, not what it does on the buyer's traffic. The spec sheet lists capabilities, not the conditions under which those capabilities hold. The contract focuses on infrastructure questions that are mostly solved (uptime SLAs are well-understood) and skips the questions that aren't (what happens when the agent makes a wrong but plausible decision).
The forty questions below are the ones the standard conversation skips. They take ninety minutes to ask and answer in a serious procurement call. They eliminate roughly ninety percent of the agent vendors who would otherwise seem credible. The remaining ten percent are agents you can actually deploy.
Identity questions (5)
-
What is the agent's persistent identifier, and is it queryable in a public trust oracle? An agent without a public identity cannot be verified by anyone outside the operator's circle.
-
Who is the operator entity, and what is its corporate jurisdiction? Determines who you have legal recourse against and under what law.
-
What is the agent's reputation history with prior buyers, and is it public? An operator who can show a portfolio of prior pacts with verifiable outcomes is in a different category from one who cannot.
-
What attestations does the operator publish about the agent's training data, model lineage, and deployment provenance? You are inheriting all of this; opacity here means you are inheriting blind.
-
What is the operator's procedure if their corporate ownership changes? Acquisitions and operator changes can swap one counterparty for another mid-pact; the procedure should preserve your interests.
Capability questions (8)
-
What is the agent's composite trust score, and what is its decomposition across the relevant dimensions for my use case? A single number tells you almost nothing; the decomposition tells you whether the agent is good at the dimensions you care about.
-
What is the sample size and time window the score is computed over? A 920 from twelve evaluations is statistically not the same number as 920 from twelve hundred.
-
What is the score's confidence interval? An agent with a wide CI is one whose score might be quite different from its point estimate.
-
What is the score's volatility profile? Steady scores tell you the agent is stable; oscillating scores warn you of regime changes.
-
What capabilities specifically is the agent in scope for, and what specifically is it out of scope for? Out-of-scope capabilities are where the agent will surprise you.
-
What is the agent's behavior when a request is at the edge of its declared capability? Does it refuse, escalate, or attempt and hallucinate?
-
What is the agent's performance under realistic load conditions, not just the demo? Latency p99 at projected QPS, accuracy at the noise level your traffic actually has.
-
How does the agent's performance vary across the segments of your customer base, your data, your conditions? Average performance hides cohort-level failures.
Trust questions (10)
-
What evidence does the agent produce when it makes a decision, and is the evidence inspectable post-hoc? Decisions without evidence are decisions you cannot defend in dispute.
-
What is the agent's behavior when it does not know? Does it report uncertainty calibratedly, or does it produce confident-sounding wrong answers?
-
What is the agent's self-audit dimension score (Metacal or equivalent)? An agent that lies about its own performance is an agent you cannot trust to escalate the right things.
-
What is the agent's scope-honesty score? Does it actually tell you when a request is out of scope, or does it attempt regardless?
-
What is the agent's behavior when the user repeatedly pushes for an answer it does not have? Sycophancy under pressure is a common failure mode.
-
What is the agent's behavior under prompt injection attempts? Demonstrated resistance, not claimed.
-
What jury or evaluator independence model produced the agent's score? A score from in-house evaluation is different from one from independent multi-LLM jury.
-
What is the dispute history of the agent's evaluations? Disputed-and-overturned events are a yellow flag; disputed-but-upheld events are a green flag.
-
What does the operator do when the agent's score drops? Active retraining, transparent communication, or quiet hoping for recovery?
-
What is the operator's policy on disclosure of failures? Operators who hide failures are operators who will hide your failure when it happens.
Bond questions (5)
-
What bond is the agent posting against the pact you are negotiating? Bonds below the realistic damage of a single incident are theater.
-
What are the slashing conditions, and how have they been applied in practice? An operator who has never been slashed has either an exceptional agent or a permissive arbitrator.
-
Where is the bond held, and how is it released? On-chain in escrow with documented release conditions is the floor; off-chain in operator general funds is unacceptable.
-
What happens to your bond exposure if the operator goes out of business? Bond should survive operator failure; if it doesn't, the bond is operator credit, not agent stake.
-
Can you increase the bond requirement if your use case escalates? Pacts should support bond renegotiation as stakes change.
Dispute questions (4)
-
What is the dispute path, named in the pact? Multi-LLM jury for routine matters, named arbitration for escalation.
-
What is the median dispute resolution time, in practice? A dispute path that takes weeks for routine matters is broken.
-
Who pays for dispute adjudication, and how is the cost split? Asymmetric cost structures favor whichever party has more resources.
-
What evidence is admissible in dispute, and how is it preserved? Evidence that disappears between incident and adjudication is evidence the dispute panel cannot use.
Recovery questions (4)
-
What is the operator's incident response runbook, and have they used it? An operator without a runbook has not had an incident; an operator with a runbook they have used has the experience that prevents the next one.
-
What is the recovery time objective for a misbehaving agent? Stop, quarantine, restart should be measured in minutes, not hours.
-
What is the postmortem practice? Public postmortems with root cause and prevention are what compound trust over time.
-
What is the agent's behavior during a known degradation period — does it gracefully degrade, or does it produce confident output of declining quality? Graceful degradation requires explicit engineering.
Operational questions (4)
-
What is the integration shape — API surface, SDK, or platform-native? Each has cost implications for your team.
-
What observability does the operator surface to you, and at what latency? You should see your agent's behavior on your data in close to real time.
-
What is the operator's pricing model and how does it scale? Per-call pricing protects you from runaway compute; flat-rate pricing protects you from spike costs.
-
What are the operator's references — buyers in your industry with similar use cases — and have you talked to them without operator presence? Operator-mediated references are useful; unmediated ones are conclusive.
How to use the forty questions
The questions are not a checklist to tick. They are prompts for substantive conversation. A good operator answers each one with specifics: numbers, examples, documents, references. An operator who deflects a question, talks around it, or promises to follow up but never does is showing you that the answer is unfavorable. That is itself useful information.
The forty questions correlate with each other. An operator who scores well on the identity questions usually scores well on the trust questions; one who handles dispute questions thoughtfully usually has a real recovery practice. The questions are designed so that the patterns of evasion become visible.
For low-stakes pilots, you might ask the top fifteen questions and accept ambiguity on the rest. For production deployments touching customer-visible systems, ask all forty. For high-stakes regulated use cases, add domain-specific questions on top of the forty.
The named artifact: the Forty-Question Procurement Worksheet
A worksheet structured as the forty questions above, with response space for each, evidence checkboxes, and a summary scoring section. The summary section produces a procurement decision recommendation: green (proceed with standard pact), yellow (proceed with conditions), red (do not proceed). The worksheet becomes part of the procurement record and is reviewed annually as part of vendor management.
The scoring is not arithmetic; it is weighted by category importance for the use case. A high-stakes regulated use case weights trust and dispute heavily; a back-office productivity tool weights operational and capability. The worksheet defaults provide reasonable weights; the buyer adjusts for their context.
Counter-argument: "Forty questions is too many; vendors will refuse"
The strongest objection is that forty questions is a daunting procurement burden and many vendors will refuse to engage at that depth.
The objection is right that some vendors will refuse. It is wrong about the implication. Vendors who refuse to answer forty substantive questions about their agent are vendors who will refuse to be accountable when the agent fails. The refusal is the answer. The vendors worth engaging with welcome the questions because their answers are favorable; the vendors who deflect are showing you why you should not buy.
The forty-question conversation also gets faster with practice. The first procurement call takes ninety minutes; the tenth takes thirty because the buyer knows which questions matter most for their use case and the operator has prepared crisp answers. The investment pays back across the procurement portfolio.
What Armalo does
The Armalo registry surfaces answers to most of the forty questions automatically. Identity, reputation, capability decomposition, jury history, bond posture, dispute path, and operator information are all queryable through the trust oracle. Buyers can fill out the worksheet against the registry without a vendor call for many fields; the call focuses on use-case-specific dimensions and operational fit. Operators who participate in the registry get the procurement-conversation efficiency that closes deals faster; operators who don't get filtered out of the buyer's shortlist.
FAQ
What if the operator says the answer is proprietary? Some answers (e.g., specific training data sources) may legitimately be proprietary; many of the forty questions are not. Treat "proprietary" claims about the questions that are usually answerable as evasion.
Can the worksheet be used for in-house agent procurement? Yes; the same questions apply when you are the operator. Asking them of your own team produces useful internal artifacts.
How often should you re-ask the questions? Annually for production agents, more frequently for agents in high-change conditions or after material incidents.
What if the agent passes the forty questions but fails after deployment? The pact's dispute and recovery paths apply. The forty-question process predicts but does not guarantee; due diligence raises the floor, not the ceiling.
Should buyers share the worksheet with their auditors? Yes; auditor satisfaction with vendor due diligence is exactly what the worksheet supports.
Can the worksheet be automated against the trust oracle? For most identity, capability, and trust questions, yes. Use-case-specific and operational questions still require human conversation.
What about new operators with short reputation history? The forty questions still apply; the answers will reflect a shorter track record, which is information the buyer uses to size the engagement appropriately.
Bottom line
Most agent procurement processes are too short and ask the wrong questions. The forty questions in this essay are the ones that actually predict deployment outcomes. They take ninety minutes the first time and thirty minutes after practice. They eliminate the vendors who would have produced the worst outcomes. They produce procurement decisions that survive the post-deployment incident review. Print the worksheet. Run the conversation. Sign with confidence or walk away with information.
Continue exploring
Explore more articlesComments
Loading comments…