Almost every AI announcement now comes with a chart: a new model tops a leaderboard, an agent beats a benchmark, a score rises by a few points. In October 2026 several research groups published a more uncomfortable message. AI agent benchmarks often measure something narrower than the headline suggests, the best agents still finish less than half of realistic multi-step work, and a single pass/fail number can hide both real progress and real risk. At the same time, banks such as Santander and TD described how they test agents on their own tasks before letting them near customers. Read together, these publications amount to a practical guide to AI agent evaluation for anyone deciding which agent to deploy.
What an AI agent benchmark actually measures
An AI agent is not just a model. A model is wrapped in a scaffold (also called a harness): the software that manages state, exposes tools and turns the model's output into actions. Stanford researchers make this point precisely: the model and the scaffold together form the system, and an evaluation has to decide which of the two it is trying to measure.
That distinction sounds academic until you buy something. A vendor demo, a public leaderboard and your own production setup almost never use the same scaffold, the same tools or the same tasks. A score is only useful if you know what conclusion it can support.
What October 2026 research found
1. Leaderboards rank systems well, but models much less reliably
In Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard, researchers from Stanford University and the University of Illinois Urbana-Champaign analysed rollouts from 22 benchmarks drawn from the Holistic Agent Leaderboard (HAL) and the Harbor Index. Their findings are directly relevant to buyers:
- Reliability depends on the claim. Fixed model–scaffold systems were ranked reliably (reliability of 0.935 to 0.994), but rankings of the underlying models were much less reliable (0.148 to 0.841).
- The scaffold can change the conclusion. Across benchmarks, how well different scaffolds preserved the same model ranking ranged from 0.151 to 0.852. On the weakest benchmark, changing only the scaffold could substantially change which model looked best.
- More tasks do not fix everything. When uncertainty comes from limited scaffold coverage, even infinitely many similar tasks would improve model-ranking reliability by at most about 0.10.
- Diversity beats volume. At the same task budget, spreading evaluation across a range of benchmarks raised projected ranking reliability from about 0.44 to 0.75, and a reliability-aware allocation could cut projected evaluation cost by up to 83%.
The authors' recommendation is simple: decide first what you want a score to mean, then spend your evaluation budget on the sources of uncertainty that matter.
2. Realistic workflows: best agents complete less than half the work
On 19 October 2026 Stanford HAI summarised three years of building progressively harder medical AI benchmarks. Language models score near-perfect on licensing exams, but in PhysicianBench, 100 long-horizon tasks executed inside an electronic health record sandbox, the best of 12 models (GPT-5.5) completed 46% of tasks on a single attempt and only 28% in all three of its runs. In HealthAdminBench, 135 administrative tasks such as prior authorizations and appeals, the best agent completed 36.3% end-to-end.
Two details matter far beyond healthcare. First, GPT-5.4 passed 82.8% of individual subtasks but completed only 26.7% of whole tasks: competence on the parts does not add up to reliability across a workflow. Second, among action-execution failures, 62% for GPT-5.5 and 73% for Claude Opus 4.6 were cases where the agent wrote the correct action in its note but never actually placed the order. The Stanford team concludes that the dominant failures come from carrying information across steps, documents and systems, pointing to agent architecture and memory, better harnesses, as much as to bigger models.
3. Long-horizon agents: partial credit and premature "done"
Long-Horizon-Terminal-Bench 2.0, from Tencent, the University of Maryland, Carnegie Mellon University and other universities, runs 77 tasks across 13 domains for up to three hours, with a verifier checkpoint every 30 minutes. The strongest model, GPT-6-astra, resolved 48.1% of tasks. On 30 reconstruction tasks, binary pass/fail scores averaged 0.221 while dense progress scores averaged 0.653, so outcome-only grading discarded a lot of useful intermediate work.
The time dimension is just as revealing. 42% of solves arrived after the first hour, and a one-hour budget would have reordered the middle of the leaderboard. Across all models, 92% of completion claims occurred on runs that never passed verification. Cost is uneven too: estimated spend for the full suite ranged from about $285 to about $30,581 depending on the model, and most money went to tasks that were never solved.
4. Enterprise readiness needs more than one number
Salesforce AI Research released EGI-Bench, which combines eight enterprise-facing agent benchmarks (including CRM operations, enterprise search, MCP tool use and long-context software engineering) into six dimensions: task outcome, tool actioning, planning, retrieval grounding, safety and abstention, and engineering quality. Its top-line score ranged from 39.6 to 65.6 across fourteen models, and no single model led every dimension. Planning was the weakest dimension for every model, topping out at 30.89. The authors also allow safety and tool-use floors to block a release regardless of the overall score, so a single aggregate cannot hide a regression that matters.
How banks are evaluating AI agents in practice
Two October announcements show what disciplined evaluation looks like inside a bank.
Santander said on 19 October 2026 that its AI Lab, working with NVIDIA, has built Model Arena to evaluate open-weight models against banking tasks and pick the right model for each workload, and is developing it to design, simulate, evaluate and supervise banking agents, including adversarial testing of conversational AI in controlled environments. By moving from large general-purpose models to smaller specialised ones, the bank reports cutting serving costs in selected experimental use cases from around €5 per interaction to about €0.03, and in one experimental customer-service case it improved conversation quality while reducing costs by more than 50%.
TD Bank U.S. launched Credit Assist on 22 October 2026, a multi-agent capability that produces a cited first draft of six credit memo sections in under 30 minutes instead of up to three working days. Before launch it was tested on 220 curated deals with input from more than 70 domain experts, and 95% of evaluated outputs passed subject-matter expert review. Credit professionals remain responsible for lending recommendations and final decisions.
Neither bank relies on a public leaderboard. Both evaluate on their own tasks, with their own experts, against their own standards, and both keep humans accountable for the decision.
An AI agent evaluation playbook for technology leaders
When a team or vendor proposes an agent, I now use these questions:
- Evaluate the system you will deploy. Test the exact model, scaffold, tools and prompts that will run in production. A model's leaderboard rank says little about your configuration.
- Build a benchmark from your own work. Curate real cases, as TD did with 220 deals, and have domain experts define what "good" means. Synthetic data is fine for privacy; invented tasks are not.
- Check actions, not just text. Grade the state of the system after the run: was the payment created, the ticket updated, the order placed? Stanford's results show agents describing the right action without doing it.
- Measure end-to-end and step-level results separately. High subtask scores can coexist with low workflow completion. Track both, plus partial progress for long tasks.
- Measure consistency. Run important scenarios several times. A task solved once in three runs is not a capability you can put in front of customers.
- Never trust "done" without verification. Put an independent check between an agent's completion claim and anything irreversible.
- Set floors for safety and tool use. Like EGI-Bench's release gates, a strong average must not hide a weak dimension such as abstention or policy compliance.
- Track cost per completed task. Include failed runs. Santander's results show that smaller, specialised models can change the economics entirely.
These checks build on the engineering basics I described in AI agents in production: narrow tools, deterministic guardrails and trajectory tests. When agents start moving money, evaluation becomes part of authorization; in Agentic payments in 2026 I look at how banks and card networks decide what an agent is allowed to do. And if specialised models are the goal, self-hosting LLMs explains the serving and cost trade-offs.
Closing thought
The most useful AI research this month did not announce a bigger model. It showed that our measuring instruments for agents are still immature: scores depend on the scaffold, pass/fail grading hides progress, and confident "done" messages are often wrong. For technology leaders that is good news in disguise. The organisations that build their own evaluation discipline, on their own tasks, with their own experts and clear release gates, will deploy agents faster and more safely than those waiting for the next leaderboard.
References
- Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard, M. Hardy, R. Azam, A. Reuel, M. Kochenderfer, S. Koyejo (Stanford University, University of Illinois Urbana-Champaign), arXiv:2610.00651, October 2026
- Measuring What Medical AI Can Actually Do, Stanford HAI, 19 October 2026
- Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress, Z. Li, Z. Li, Y. Shi et al. (Tencent, University of Maryland, Carnegie Mellon University and others), arXiv:2610.14146, 9 October 2026
- EGI-Bench: An Enterprise General Intelligence Benchmark Suite, Z. Liu, Z. Lan et al. (Salesforce AI Research), arXiv:2610.27234, October 2026
- Santander builds AI capabilities that improve performance and lower costs using NVIDIA AI infrastructure, Banco Santander, 19 October 2026
- TD Launches Credit Assist, Turning Days of Commercial Credit Work into Minutes, TD Bank U.S., 22 October 2026