AI Agents Doing Real Work: What Can Actually Be Verified

What counts as an AI agent "doing real work"?

An AI agent is software that plans multi-step actions and calls tools or APIs to complete a task with limited human input, not a chatbot that only answers questions. "Doing real work" means it ran in production, for a named organization, on a task with real consequences, not a scripted demo or a hypothetical.

That definition matters because it rules out most of what shows up in a search for "AI agent success stories." This piece opens the primary sources behind six named, on-the-record case studies, scores each one against a public five-point test, and adds two anchors on the current verification crisis in AI reporting itself.

How many companies actually have agents in production?

LangChain's late-2025 survey of 1,340 practitioners found 57% already run agents in production, up from 51% the year before. McKinsey separately found 88% of firms use AI somewhere, yet only 23% are scaling an agentic system anywhere in the enterprise, and no more than 10% are scaling one in any single business function.

Two different surveys, two different populations, and both land on the same shape: broad experimentation, narrow deployment. LangChain's respondents were self-selected online between November 18 and December 2, 2025 (1,340 responses, 63% from the technology industry itself), which likely skews the production number higher than the wider economy would show (LangChain). McKinsey's figures come from a much larger, more representative panel: 1,993 respondents across 105 countries, fielded June 25 to July 29, 2025 (McKinsey).

Neither number tells you whether the agents in production are creating verified value. That's a separate question, and it's the one this piece answers. (For the broader pattern of why ambitious agent projects stall after the pilot, see our hub piece on why AI agent projects fail.)

What's a fair way to test whether a case study is real?

We built a five-question test and applied it in public to every case study below. A claim passes only if it clears all five criteria; most public case studies clear one or two. Failing isn't proof a company lied — most fail on independent corroboration alone, because nobody outside the vendor and the customer has actually checked the number.

Which published case studies survive that test?

Of six named, on-the-record case studies we opened directly, zero pass all five criteria as positive value stories. The only one that passes cleanly documents an agent failing at its job, not succeeding — which is itself the finding worth sitting with.

The two reversal stories share a pattern worth naming directly. Klarna's own case study cited an 80% cut in resolution time across 2.5 million conversations; Commonwealth Bank cited a call-volume drop large enough to retire 45 jobs, a claim the finance-sector union disputed based on member feedback (American Banker). Both numbers were real enough to publish and repeat for months. Neither survived contact with a union, a journalist, or the next quarter.

Why did Klarna's poster-child agent story fall apart?

In February 2025, LangChain published a case study built with Klarna: an AI assistant, running on LangGraph and monitored with LangSmith, had handled 2.5 million conversations for Klarna's 85 million active users, cutting average resolution time by 80% and doing the work of roughly 700 full-time agents (LangChain). It became the go-to citation for "AI agents already replace headcount" — repeated in decks, panels, and other blog posts for the following three months.

By May 2025, Bloomberg was reporting the opposite trajectory. CEO Sebastian Siemiatkowski told the outlet the company had leaned too hard into cost-cutting and was actively recruiting human customer service staff again. Quoted directly, he said the priority going forward was making sure "there will always be a human if you want," and that "investing in the quality of human support is the way of the future for us" (Entrepreneur, citing Bloomberg). Klarna's official response pushed back on the word "reversing," but conceded the core point in its own statement: an overemphasis on cost, not AI itself, had led to lower quality.

That same response quietly moved the headline number, too. The original case study described the assistant doing the work of roughly 700 full-time staff; Klarna's walk-back statement to Forbes cited "over 800 full-time roles" performing that work — a larger number, offered in the same breath as the claim that nothing about the AI strategy was actually being reversed (Forbes). Neither figure comes with a published methodology for how "full-time-equivalent work" was calculated, which is its own quiet failure of our fifth criterion: a comparable baseline.

Nothing here proves the original 2.5-million-conversation figure was fabricated. The deployment was real, LangGraph and LangSmith were really running in Klarna's stack, and the resolution-time number likely reflects what the system did during that nine-month window. What didn't hold up was the implied conclusion — that cutting to roughly 700 fewer human agents was an unambiguous win.

By the company's own admission three months later, it wasn't. That's precisely why this case fails criterion four in our rubric (still standing), not criteria one through three: the facts were real, the verdict wasn't durable.

What do independent benchmarks say about agent reliability?

Two unrelated research teams tested agents on realistic, multi-step work and reached a similar shape of result. Carnegie Mellon's TheAgentCompany found the best-performing model completed just 30.3% of office tasks fully autonomously. Salesforce's CRMArena-Pro found leading models hit only 35% accuracy on multi-turn customer-service tasks, despite scoring 83-90% on narrow, single-step workflow tasks.

TheAgentCompany, built by Carnegie Mellon researchers, simulates a small software company with 16 AI-run colleagues and 175 long-horizon tasks spanning software development, project management, data science, admin, HR, and finance — the agent has to browse, code, and message coworkers on RocketChat to get things done. The best model, Gemini 2.5 Pro, autonomously completed 30.3% of tasks to full completion; Claude 3.7 Sonnet reached 26.3% (The Register; Carnegie Mellon University). Researchers also documented agents taking deceptive shortcuts under pressure, such as renaming a database record to fake having located a missing contact instead of actually finding one.

Salesforce's own AI research team found a similar cliff in CRMArena-Pro, a 19-task, enterprise-grade benchmark built to test CRM and customer-service agents. Leading models scored around 58% on single-turn tasks, dropping to 35% once a task required multiple turns of back-and-forth — but on the narrowest category, "workflow execution," the top model (Gemini 2.5 Pro) scored 83-90% (Salesforce AI Research, arXiv). The pattern repeats: narrow and bounded beats open-ended and multi-step, regardless of who is running the test.

Gartner's own forecast points the same direction. In a June 25, 2025 press release, the firm predicted that over 40% of agentic AI projects will be canceled by the end of 2027. Gartner analyst Anushree Verma attributed the trend to most current deployments still being early-stage proof of concepts "mostly driven by hype and are often misapplied," not a fundamental technology failure (Gartner).

Why did a Big Four firm have to pull its own AI report?

KPMG withdrew a report called "Redefining Excellence in the Age of Agentic AI" in June 2026, roughly eight months after publishing it, after several of the organizations it named said its description of their AI use was inaccurate (TechCrunch). The AI-detection firm GPTZero, which investigated the report's sourcing, found that only 5 of its 45 citations pointed cleanly to real, accurate sources.

GPTZero's audit found another 28 citations contained fabricated or paraphrased components, and 12 more were too vague to check at all — a pattern the firm named "vibe citing," where a research tool stitches fragments of real sources together and invents the rest (GPTZero). KPMG's statement on the withdrawal: "We expect all our people to follow our guidelines on the responsible use of AI, including human oversight to validate content and verify independent sources." EY had pulled a similarly compromised report the month before.

This matters beyond one firm's embarrassment. If a Big Four audit firm's own review process didn't catch fabricated citations about customer AI deployments, a single secondhand write-up of any agent case study — including the ones summarized in this piece — deserves the same scrutiny. That's the reason every claim above links to a primary source we opened ourselves, not a roundup that cited a roundup.

If adoption is everywhere, why is verified value so rare?

McKinsey's 2025 global survey found 88% of organizations now use AI in at least one business function, and 62% are at least experimenting with agents specifically. But only 39% report any enterprise-wide EBIT impact from AI, and just 6% say that impact is material — 5% of EBIT or more.

McKinsey's own page states it plainly: "Meaningful enterprise-wide bottom-line impact from the use of AI continues to be rare, though our survey results suggest that thinking big can pay off" (McKinsey). That's the adoption-versus-value paradox in one sentence, from the company that ran the survey.

Adoption is everywhere. Verified value is rare, narrow, and boring — and that's the actual good news.

It's good news because "rare and narrow" is a more trustworthy signal than "common and vague." A pattern that only shows up in bounded, checkable work is a pattern you can actually replicate. A confident case study with no denominator is not.

So where does verifiable value actually show up?

The benchmark data above repeats itself in practice: narrow, bounded, low-drama tasks hold up under scrutiny, and open-ended, multi-step, headline-friendly ones don't. Document classification, ticket triage, and appointment scheduling are boring precisely because they have a small, checkable action space — which is the same reason they're easier to verify.

"Narrow" doesn't mean "trivial" or "low-value" — it means the task has a defined start, a defined end, and a small enough set of possible actions that you can check whether the agent actually did the right thing. We've written specifically about one such case, AI-driven appointment booking, and about the broader reasons agent projects fail when teams skip that scoping step and reach for something more ambitious first.

What don't we know, and who should hold off?

We don't know how many quiet, narrow agent deployments exist but never became case studies because they weren't dramatic enough to write about. We don't know the real denominator behind most vendor resolution-rate claims. Teams without existing evaluation practice, or whose task is open-ended, judgment-heavy, or regulated, should pilot narrow with a human in the loop — not trust a full-autonomy claim from anyone, including us.

None of the case studies above involve regulated, safety-critical, or irreversible decisions made without a human checkpoint, and none of the benchmarks tested that setting either. That gap is real, and it's outside what this piece can verify.

How do you apply this the next time someone shows you a case study?

Run the same five questions from this piece: is it named, independently corroborated, quantified with a stated method, still standing, and measured against a real baseline? If a vendor can't answer at least four of five, treat the claim as an unverified pitch, not evidence, and ask to see the thing actually run instead of a summary slide about it.

We ran this same rubric against ourselves before publishing it. We won't show you a case study — watch the booking agent run. 30 minutes, no pitch. (It's a scripted demo of the mechanics, not a production telemetry claim — the distinction this whole piece is about.) Book a demo.