How Do You Know an AI Agent Works Before You Pay?

What does "done" mean for an AI agent?

For ordinary software, "done" means the code passes a fixed set of tests: same input, same output, every time. An AI agent is probabilistic. The same request can succeed on Monday and fail on Tuesday. So "done" cannot be a checkbox. It has to be a measured pass rate on a set of real cases, agreed before launch.

This is not a small distinction. Gartner predicts that "over 40% of agentic AI projects will be canceled by the end of 2027", citing escalating costs, unclear business value, and inadequate risk controls. A vague definition of "done" is how a project ends up in that 40%. You pay, the demo dazzles, and then the agent meets real users.

Practitioners have started to name the fix. Rick Pollick puts it plainly: "Acceptance is no longer a checkbox; it is a threshold on a distribution". Wavect's AI MVP scope template makes the same move, insisting that "it works" is not testable for LLMs and must be replaced by a measured target. Both are right. This article takes their engineering insight and turns it into a buyer's tool: clauses you can put in a contract before money changes hands.

What is an acceptance eval?

Acceptance evals are a pre-agreed test suite the agent must pass, at a pre-agreed rate, before final payment. You freeze a set of real cases, write down the correct outcome for each, set a passing threshold, and make the last invoice depend on hitting it. They turn "the demo looked great" into a number both sides signed, before either side is emotionally or financially committed.

Notice the gap most teams leave open. In LangChain's State of Agent Engineering report (1,340 responses, surveyed 18 November to 2 December 2025), 89% of organizations had some form of observability, but only 52.4% ran offline evaluations on test sets. Fully 29.5% reported not evaluating their agents at all; even among teams already running agents in production, 22.8% still ran no evals.

Read that contrast slowly. Nearly everyone watches their agent in production. Far fewer test it before shipping. Observability tells you the plane is on fire. An acceptance eval keeps it on the ground until it is airworthy. As a buyer, you want the gate, not just the smoke alarm.

Why does passing the demo once mean nothing?

Because one success measures luck, not reliability. A probabilistic system can produce a correct answer on the first try and a wrong one on the fifth, from the identical prompt. A live demo is a single sample. To trust an agent you need to know how often it is right across many independent attempts on the same task, not whether it happened to shine while you watched.

Researchers made this measurable in 2024 with tau-bench, a benchmark that makes an agent hold a real conversation and follow written policies while calling tools. They added a metric called pass^k: the share of tasks the agent gets right on all k independent attempts. Their finding, verbatim: state-of-the-art function calling agents (like gpt-4o) "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."

Read that carefully, and read its limits. On the retail tasks they tested, with the models they tested, a leading agent got the job right on all eight tries fewer than a quarter of the time. Those numbers describe that 2024 benchmark, not your agent, and they do not transfer to your use case. But the lesson does: a single green demo tells you almost nothing about consistency.

Here is where our approach differs from the common templates, and it is a friendly disagreement. Pollick's "threshold on a distribution" and Wavect's template both set a passing rate on an eval set, ideally averaged across several runs. That average is a mean, and a mean can hide an agent that is a coin flip on the same input. So we add a second, stricter clause: repeated-trial reliability, measured pass^k-style on your highest-risk cases, where the agent must succeed on every attempt, not on average.

What should an acceptance eval measure?

Measure six things, because a booking that is fast, cheap, and confidently wrong is still wrong. Cover task success on a single attempt; repeated-trial reliability on the same input; correct refusal and escalation when the agent should not act; safety, meaning no harmful or policy-breaking output; latency your users will feel; and cost per completed task. Each becomes a clause with its own threshold.

The point of splitting them is leverage. A vendor can quietly trade safety for speed, or accuracy for cost, if "done" is one blurry number. Separate clauses stop that. They also map cleanly onto a statement of work, which is where the next section takes them. Building this measurement discipline in from the start is the core of a production-ready agent framework, not an afterthought bolted on before launch.

What do the acceptance clauses look like as a contract?

They look like numbered SOW clauses, each pairing a test with a threshold you own. Below is a copy-paste starting template for an appointment-booking agent. It is written the way a contract reads: a clause, what it verifies, an illustrative threshold, and how you measure it. Adapt every row to your own risk before you sign anything.

Every number in that table is an illustrative starting point that you set from your own risk tolerance. None is an industry benchmark, and none is drawn from tau-bench or any other study. The pass^8 target in clause 5.2 borrows only the idea of repeated-trial measurement from that paper; a narrow, well-specified booking task is not the open-ended retail benchmark those researchers used, so their figures do not carry over.

A worked example makes the clauses concrete. Say your booking agent, described in our appointment-booking guide, faces a frozen set of 120 real requests. Clause 5.1 asks for 95% task success, so it may miss at most 6 of the 120. Clause 5.2 pulls the 30 riskiest cases, the reschedules and double-bookings, and demands pass^8 of 90%: the agent must get all eight independent runs right on at least 27 of those 30. An agent that averages well but flakes on the same hard reschedule fails 5.2, exactly as it should. That single clause is your whole thesis, priced.

A caveat you must respect: this table is a starting template, not legal advice. Have your own counsel adapt the wording, the remedies, and the payment triggers to your jurisdiction and contract.

What do you ask a vendor that has no evals?

Ask five questions, and watch how fast the room changes. Can you show me your eval set and the golden answers you score against? What is your pass rate on it, and your pass^k on the risky cases? Who owns that set, and can we add our own cases? What happens to the final invoice if you miss a threshold? And how do you catch regressions when you change the model?

A serious vendor answers all five without flinching, because they already run this discipline internally. A vendor selling a demo will reach for adjectives: robust, intelligent, enterprise-grade. Those are not answers. Insist on the set, the numbers, and the payment consequence in writing.

There is a blunt way to say why this matters:

If your vendor can't show you their evals, you are the eval.

That is not a slogan, it is a cost. If no one tested the agent on a frozen set before you paid, then your customers are the test set, your production traffic is the eval run, and your brand absorbs the failures. Pollick's team frames the same duty from the delivery side: the golden set "cannot be owned by engineering alone", because deciding what a good answer looks like is product and risk work. As the buyer, you are the product and risk owner. Own the set.

When are acceptance evals overkill?

When the stakes are low, the work is reversible, and a wrong answer costs nothing. A first-draft summarizer for your own eyes, a brainstorming helper, a throwaway prototype you will rebuild next month, none of these justify the cost of writing and maintaining a frozen eval set. Rigor should match risk. Over-engineering a low-stakes tool wastes the same money a missing eval would have lost.

The line is simple. If a wrong output can cost money, break safety, or damage trust with a customer, write the clauses. If the worst case is a slightly awkward draft you will edit anyway, skip them. The 29.5% of teams running no evals are not all wrong; some of them are summarizing internal notes. The danger is booking real appointments, moving real money, or making promises to real customers with a summarizer's level of care. That mismatch is how projects join Gartner's canceled 40%.

Ready to define "done" before you sign?

We do not negotiate on standards, and neither should you. At Visione Edge we build agents against acceptance evals from the first day, so "done" is a number we both agree on and the final payment depends on hitting it. If you are evaluating a vendor, or scoping your own build, we will help you write the clauses and the eval set that protect you.

Book a call and bring the agent you are about to pay for. We will show you exactly how we would test it, and where its numbers would land. If you want the full picture first, start with our guide to a production-ready agent framework.