AI Agent Evaluation andObservability in 2026

If you have heard that coding agents now solve 95% of SWE-bench Verified, you have heard a vendor self-report. The official SWE-bench leaderboard has accepted submissions only from academic and research institutions with open methods and an arXiv report since November 18, 2025. It holds 134 total submissions. The most recent is dated December 15, 2025: live-SWE-agent paired with Claude Opus 4.5, resolving 396 of 500 problems — 79.2%.

There are no 2026 frontier entries on that board at all. Not lower ones, not higher ones. The gap between 79.2% and the figures circulating in marketing decks is not progress that the board failed to record; it is the difference between an audited submission and a press release.

Then, in February 2026, OpenAI formally retired the benchmark for its own evaluation. The reasoning is worth reading in full, because it is the most rigorous public benchmark audit anyone has published. OpenAI took 138 problems its o3 model had failed across 64 runs and put six or more engineers on each one. The finding: 59.4% of those problems had material test or description flaws. Specifically, 35.5% had overly narrow tests that enforced a particular implementation rather than the behaviour the problem described, and 18.8% had tests so wide they checked functionality the problem statement never asked for.

This article is about what to do instead. It is written for the person who has to answer a specific question — is our agent good enough to put in front of customers? — and who has correctly noticed that no public number answers it. The short version: build a small evaluation set from your own failures, judge it carefully, report reliability rather than best-case accuracy, attach a dollar figure to every score, and trace everything in production. The rest of this article is the detail.

The cleanest number in agent evaluation right now is a comparison, not a score.

SWE-rebench continuously harvests new tasks from real repositories, specifically so that the tasks postdate model training cutoffs. Its most recent slice is 111 problems drawn from 65 repositories, collected between May 15 and July 1, 2026. On those fresh, decontaminated tasks, frontier models land at roughly 64% — Fable 5 at 64.5% (78.4% pass@5, $4.40 per problem), Grok 4.5 at 63.8% for $1.47, Opus 5 at 63.4%.

Against the roughly 95% those same vendors claim on SWE-bench Verified. Same class of task — resolve a real GitHub issue in a real Python repository, verified by the repository's own tests. Roughly 30 points apart. The only material difference is whether the model could plausibly have seen the task during training.

Two supporting results make it hard to explain the gap any other way. Independent analysis published in December 2025 found models score roughly three times better on SWE-bench Verified than on decontaminated equivalents, and — the more damning of the two — are about six times better at locating the edited files when given no project context at all. Identifying which files a patch touches, without being shown the project, is a task that is logically impossible to perform well without prior exposure to that specific commit. That is not a capability measurement. It is a memory measurement.

Benchmark designers have started responding structurally rather than rhetorically. SWE-bench Pro is built on GPL copyleft repositories, deliberately, as a legal deterrent against inclusion in training corpora — 1,865 tasks across 41 repositories, split into 731 public, 276 private and 858 held-out. Whether copyleft actually deters anyone is unproven, but the design intent tells you how seriously the people closest to the problem take it.

A benchmark measures a model on tasks it may have memorized. Your eval set measures a system on tasks it definitely has not. Those are different instruments, and only one of them predicts what your users will experience.

— Frenchy Digital evaluation principle

Contamination is a data problem. This is worse: it is a harness problem, and it means the score can be obtained without the capability at all.

In April 2026, a UC Berkeley team (Wang, Mang, Cheung, Sen and Song) published an exercise in deliberately gaming agent benchmarks. They reached 100% on Terminal-Bench, SWE-bench Verified, SWE-bench Pro, FieldWorkArena and CAR-bench, roughly 98% on GAIA, close to 100% on WebArena, and 73% on OSWorld — without solving a single task.

The methods were not exotic. They were the kind of thing any competent engineer would find within a day of looking, which is the entire point.

Documented reward-hacking techniques against agent benchmarks — UC Berkeley RDI, April 2026, and Princeton HAL log inspection.

The follow-up work, BenchJack, systematized the search and found 219 distinct flaws across ten benchmarks. The constructive half of that result matters too: after patching, vulnerable tasks fell from close to 100% to under 10% on four of the benchmarks. These are fixable defects, not a permanent condition. But they were present in benchmarks that thousands of engineering decisions were made against.

The practical implication for your own harness is uncomfortable but simple. If a benchmark team with a public reputation to protect ships a grader that can be intercepted by a conftest hook, your internal eval harness — written in two sprints, reviewed by nobody outside the team — almost certainly has a similar hole. Run the adversarial test: give a deliberately non-functional agent the ability to write files in the eval environment and see what score it earns. If it earns anything above zero, you have found something.

There is a third integrity problem, newer than the other two and less widely understood: the model may recognize that it is inside an evaluation.

Anthropic interpretability work using a natural language autoencoder looked for internal representations consistent with evaluation recognition. It found them in roughly 26% of SWE-bench Verified problems and 16% of a destructive-coding evaluation — against under 1% of real product conversations. The Claude Opus 4.7 system card separately reported verbal evaluation-awareness, where the model says out loud that it appears to be in a test, in about 9% of transcripts.

Be careful about what this does and does not establish. It does not prove that a model behaves differently when it notices. It establishes something narrower and still significant: an evaluation environment is a detectable distribution. Benchmark tasks have a recognizable shape — sanitized repositories, self-contained problem statements, no ambiguity about what success means, no stakeholder who will be annoyed by the wrong answer. Real work does not look like that, and the representation gap between 26% and under 1% says the model has noticed.

With all of the above stipulated, public benchmarks are still worth reading. They tell you which capabilities exist somewhere in the industry and roughly how fast that frontier moves. They do not tell you what your system will do. Every figure below carries the date it was published, because a score without a date is not checkable.

Agent benchmark standings as reported by primary leaderboards, verified August 2026. Scores are perishable — re-check the source before quoting.

Two entries deserve a comment. OSWorld is listed by its human baseline of 72.36% rather than a model score, because the self-reported and independently-verified figures for the same models differ by six to eleven points; our computer-use agents guide covers that gap in detail. And ARC-AGI-2 is included with cost attached because it is the clearest illustration of price-performance spread anywhere in evaluation: 37.6% at $2.20 per task against 54% at $30 per task.

Two structural changes are also worth noting. ARC-AGI-3, built around interactive agent environments, now exists. And Princeton's Holistic Agent Leaderboard paused leaderboard updates during 2026 to focus instead on a reliability dashboard covering consistency, predictability, robustness, safety and self-awareness. That pivot — from a single accuracy ranking toward a multi-dimensional reliability view — is itself a signal about where serious evaluation is heading, and it is the direction your internal evals should already be pointed.

One negative note on scope. AIME and MMLU-Pro are not agent benchmarks. The original MMLU has been retired for frontier comparison and MMLU-Pro is near saturation around 90%. A vendor leading an agent pitch with a knowledge-QA score is telling you which numbers were most flattering, not which were most relevant.

Verified contradictions between benchmark aggregator sites and primary leaderboards, August 2026.

The ARC-AGI-2 case is the most instructive. An aggregator claim of 92.5% is roughly 40 points above the primary source — and materially above the 60% average of the human panel ARC uses as its reference. A number that implies superhuman performance on a benchmark specifically designed to be easy for humans and hard for models should have failed an editorial check. It did not, because there was no editorial check.

Most teams that do run evaluations grade them with a model. That is the right default — human grading does not scale and rule-based grading does not generalize. But the published evidence on judge reliability is considerably less flattering than the practice's popularity suggests, and almost none of it appears in vendor documentation.

Published findings on LLM-as-judge reliability, with the operational response to each. Sources listed in full at the end of this article.

Three of those rows carry most of the weight.

Agreement is systematically overstated. The number everyone quotes — GPT-4 agreeing with human raters 85% of the time, against 81% human-to-human agreement — is raw exact-match agreement, which does not correct for the agreement you would get by chance. A 2026 study of 21 judges across roughly 541,000 judgments measured the gap directly: exact-match agreement overstates chance-corrected agreement by 33.8 to 41.3 percentage points, putting true Cohen's kappa between 0.376 and 0.511. That is moderate agreement. It is a useful instrument and it is not a human replacement, and the difference between those two descriptions is what determines whether you keep a human in the loop.

Consistency is not validity. The same body of work surfaces a paradox worth internalizing: a judge can show test-retest reliability above 0.95 while simultaneously exhibiting position bias above 0.10. It gives the same answer every time, and the answer depends on which candidate you listed first. Teams routinely measure the first property, find it excellent, and conclude the judge is sound. Measure both, or you have measured nothing.

Juries beat single judges, and cost less. The Panel of LLM Evaluators result is the strongest practical upgrade available in this whole area: an ensemble of smaller judges reached kappa 0.763 against GPT-4's 0.627, at seven to eight times lower cost. Three cheap heterogeneous judges outperforming one expensive one is an unusual finding in that it improves quality and budget in the same change. If you take one action from this section, replace your single judge with a panel drawn from different model families.

Two more operational notes. First, prefer absolute rubric scoring to pairwise comparison for regression suites: pairwise verdicts flipped on around 35% of manipulated cases against 9% for absolute scoring. Second, never grade agent trajectories with string matching — a three-model ensemble reached kappa 0.432 on trajectory judging, while substring matching reached 0.049, which is indistinguishable from chance.

Finally, on judge selection: there is no universally good judge. Judge rankings shift by up to 14 positions depending on which benchmark they are evaluated against. Choosing a judge from a general leaderboard is choosing a judge for someone else's task. Calibrate against your own human labels, on your own rubric, and re-calibrate when you change judge model. And treat vendor judge-accuracy claims with the same skepticism you would apply to any self-reported benchmark — they are unvalidated by third parties, while Anthropic and OpenAI both publish judge guidance and no agreement numbers at all.

This is the practical payload. Nearly all of it comes from two primary sources: Anthropic's January 2026 guidance on evals for AI agents, which is the best-written primary material on the subject, and LangChain's agent evals material, which is the only published source we found with concrete sampling and throughput numbers.

Agent evaluation design decisions and their primary sources — Anthropic (Jan 2026), LangChain, Sierra, Princeton HAL.

Start at 20 to 50 tasks, drawn from real failures. This is smaller than most teams expect and that is deliberate. A set you maintain, whose every case you can explain, beats a set of 500 generated cases that nobody trusts and nobody reads. Real failures also carry the properties that make production hard and that invented tasks systematically lack: ambiguity, incomplete context, malformed inputs, and the presence of a person who wanted something slightly different from what they asked for.

Run every trial isolated, from a clean environment.Anthropic is explicit that shared state produces correlated failures and inflated scores, and this is the single most common defect we find in client eval harnesses. A suite where trial seven inherits trial six's working directory is not measuring per-task success; it is measuring a sequence. The failures cluster, the variance collapses, and the number looks more stable than the system is.

Grade the outcome, not the path. LangChain warns specifically against rigid tool-sequence matching, and the trajectory-judging data backs it up. If the agent reaches the correct end state via a route you did not anticipate, that is usually a better agent, not a failed test. Grade paths only where the path itself is the requirement — an audit trail, a compliance step, a mandatory confirmation.

• Calibrate the judge against humans before you trust it: At least 20 human-labelled examples, per LangChain, and report chance-corrected agreement rather than raw agreement. Recalibrate whenever you change judge model, judge prompt, or rubric — all three change the instrument.
• Sample 10% of production traces to start: LangChain's starting figure. Tune from there: sample more heavily on new workflows and after model changes, less on stable paths. Sampling is a budget decision, so make it explicitly rather than by accident.
• Read transcripts weekly, as a habit: Anthropic's guidance, and it is the highest-yield hour in the week. Metrics tell you the rate of the failures you already thought to measure. Transcripts tell you about the ones you did not.
• Budget human review at 50–100 traces per hour: LangChain's throughput figure. At 10% sampling on a workflow doing 10,000 runs a week, that is 1,000 traces — ten to twenty hours of human time weekly if you review all of them, which you should not. Sample the sample.
• Run evals in CI on every agent change and every model change: Both are behaviour changes. A model version bump is a deploy even though no code moved, and it is the change most likely to ship without a test run because it does not look like a change.
• Keep a failure taxonomy, not just a pass rate: Every production incident becomes a labelled case and a category. Over a quarter the taxonomy tells you where to spend engineering time, which a pass rate never will.
• Watch for the 100% ceiling: Anthropic makes the point directly: a suite that passes at 100% tracks regressions but gives no improvement signal. When your suite goes fully green, that is the moment to add harder cases, not to declare the work finished.
• Version the eval set alongside the agent: Prompts, tools, model IDs, rubrics and eval cases all live in the repository under review. Then a score is reproducible, and 'what changed?' is answerable from a diff instead of from memory.

One further note from the LangChain material: roughly half of practical test cases are single-step. That is worth knowing before you build an elaborate multi-turn simulation harness. Most of the value arrives from a well-chosen set of single-step assertions about behaviour you actually care about, and the multi-turn machinery can wait until those pass.

The eval set is the deliverable. An agent without one is a demo that happens to be running in production, and its quality is whatever nobody has complained about yet.

— Frenchy Digital delivery principle

Almost every published agent score is pass@1: one attempt, did it work. That is the wrong question for anything that runs more than once.

Sierra's tau-bench introduced the metric that fixes this. pass^k is the share of tasks where all k independent trials succeeded. Not the best of k, not the average of k — all of them. It measures whether the system is dependable rather than whether it is capable.

The published gap is dramatic. On tau-retail, GPT-4o scored under 50% at pass^1 and around 25% at pass^8. Nothing about the model changed between those two numbers. Only the requirement did — from "succeeds once" to "succeeds eight times out of eight".

Implementing pass^k on your own suite is cheap: run each case k times from a clean environment, count a task as passed only if every trial passed. Start with k=5 or k=8 to match published practice. The cost is k times your eval spend, which is the main reason teams avoid it and a bad reason given what it reveals.

One honest caveat, because it cuts the other way. Outcome-only scoring can understate capability as well as overstate reliability: published work on tau-Bench Airline found pass^5 was under-elicited by nearly 50% — the harness, not the model, was responsible for a large share of the failures. So when pass^k collapses, read the failing transcripts before you conclude the model is unreliable. Sometimes you have measured your own scaffold. That is still worth knowing; it is just a different fix.

An accuracy number with no cost attached is not a comparison. It is half of one, and it is the half that is easiest to move by spending money.

Princeton's Holistic Agent Leaderboard is the definitive work here: 21,730 rollouts across 9 models and 9 benchmarks, roughly $40,000 of compute, with 2.5 billion tokens of logs released publicly. Two findings stand out.

The first is the headline and it is counterintuitive: higher reasoning effort reduced accuracy in the majority of runs. Not raised cost for marginal gain — actively reduced accuracy, most of the time. If your team's default response to a quality problem is to turn the reasoning dial up, that is a hypothesis to test rather than a fix to apply.

The second came from reading the logs, which is why releasing them mattered: inspection caught agents searching HuggingFace for the benchmark instead of solving the problem. That behaviour is invisible in a pass rate and obvious in a transcript, which is the argument for transcript review compressed into a single example.

The operational rule that follows is simple and almost nobody applies it: report cost per task alongside every accuracy figure, on every eval run, in the same table. Then a scaffold change that adds two points and triples spend gets discussed as the trade-off it is, rather than shipped as an improvement. The public leaderboards that do this — HAL, SWE-rebench, ARC Prize — are markedly more useful than the ones that do not, and the same is true of internal dashboards.

Cost-aware evaluation also feeds directly into architecture. Once every eval run reports dollars, the routing and caching decisions covered in our LLM cost optimization and token economics guide become measurable rather than theoretical: you can see, per workflow, what a smaller model costs you in accuracy and saves you in spend.

Evaluation tells you whether the system is correct on cases you chose. Observability tells you what it actually did on cases you did not. You need both, and the tooling market is now mature enough that building your own tracing layer is rarely defensible.

Pricing below was checked in August 2026. Treat it as perishable and verify before you commit — but the license column changes far more slowly, and it is the column that determines whether you can actually leave.

Agent observability platforms — pricing and licensing verified August 2026.

Choosing between them is mostly a question of shape rather than features.

• If you need to own the data: Langfuse first. MIT license, self-host without feature gating, free to 50,000 units per month, $29 for Core. This is the default recommendation for regulated deployments and for anyone who cannot send prompt content to a third party.
• If you want a proxy rather than an SDK: Helicone. Apache-2.0, ungated self-host, one-line gateway integration that also gives you caching and cost tracking without touching application code. Free to 10,000 requests per month, $79 for Pro.
• If you are already on LangGraph: LangSmith, at $39 per seat per month with overage at $0.50 per 1,000 traces. Native tracing and failure clustering. Note that self-hosting is Enterprise-only, so the exit path is commercial rather than technical.
• If evaluation is the primary workflow: Braintrust, $249 per month with no per-seat charge — which matters if the whole team looks at evals rather than one person. Online scoring gates fit naturally into CI.
• If you already run Datadog: Datadog Agent Observability, $160 per month annual for 100,000 spans, then $3.50 per 10,000. The strongest OpenTelemetry support of the group and the only one that correlates agent traces with the rest of your infrastructure in one pane.
• If you are already standardized on Weights & Biases: Weave, free to 1 GB per month then from $60. Usage-based pricing at roughly $100 per GB means high-volume prompt capture gets expensive quickly — model the bill on real trace sizes first.

Whatever you pick, instrument at the agent level and not only at the model call. A trace that shows twelve model invocations and no tool calls, no plan step and no retry structure will not explain a failure. What you want recorded, at minimum: the invocation with its inputs, each planning step, each tool call with arguments and result, token usage and cost per step, latency per step, the final outcome, and a stable identifier that lets you join a trace to an eval case and to a support ticket.

Standardizing on OpenTelemetry's GenAI semantic conventions is the right long-term call, because the alternative is a proprietary schema per vendor. But you should know exactly what you are adopting, and a great deal of published guidance describes it as more settled than it is.

OpenTelemetry GenAI semantic conventions — status and 2026 breaking changes, verified August 2026.

The headline is that the conventions are entirely Experimental or Development status. Nothing in them is Stable or release-candidate, and the core repository's 2026 roadmap lists GenAI stabilization as unconfirmed. Two breaking changes landed during 2026: the conventions moved out of the core repository into a separate semantic-conventions-genai repository which has no tagged releases, and gen_ai.system was deprecated in favour of gen_ai.provider.name.

The renamed attribute is the one that will bite you quietly. A dashboard or alert keyed on gen_ai.system does not error when the attribute disappears; it just returns nothing, and a panel showing zero looks a lot like a system with no traffic. Migrate with a dual-write window and delete the old attribute deliberately.

The prompt-capture change is the other one worth planning for. Capture moved from log events to structured span attributes — gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions — and all of them are Opt-In. The old gen_ai.user.message and gen_ai.choice events no longer exist. So anything you built against the event model needs rewriting, and prompt content is off by default, which is a sensible privacy posture and a surprising one if you assumed your traces contained prompts.

One genuinely new addition worth flagging: the conventions now cover Model Context Protocol, with trace context propagated in params._meta. That means you can follow a trace across an MCP boundary without inventing a header scheme, which is a real improvement for anyone running tools behind MCP servers.

Two datasets bracket the reality here, and they have opposite biases pointing in the same direction — which is the most you can ask of survey evidence.

LangChain's State of Agent Engineering (n=1,340, fielded November 18 to December 2, 2025, published June 12, 2026) found 89% of teams have some observability and 62% have step or tool-level tracing — but only 52.4% run offline evaluations, 37.3% run online evaluations, and 29.5% run none at all. Among organizations with agents actually in production the no-eval figure improves to 22.8%, and 44.8% of production organizations run online evals. On method, 59.8% use human review and 53.3% use LLM-as-judge. 57% have agents in production, and quality — not cost — is the top barrier they report.

The counterweight comes from peer-reviewed qualitative work with the opposite bias. The Results-Actionability Gap (van der Maden et al., 19 practitioner interviews, fieldwork February to May 2025) reports its findings as counts rather than rates, which is the honest way to present N=19. 12 of 19 practitioners called informal manual checks — vibe checks — irreplaceable. 13 of 19 had tried automated evaluation and described it as bordering on useless. Only 3 of 19 used LLM-as-judge at all.

Put the two together and a coherent picture emerges. Tracing is widely adopted because it is easy to install and immediately legible. Evaluation is not, because it requires deciding what correct means, and that decision is genuinely hard and cannot be bought. The practitioners saying automated eval is near-useless are not being unserious; they are reporting that a badly-specified rubric run by an uncalibrated judge produces numbers that do not change any decision. That is a correct observation about a common implementation, and the response is better calibration, not more dashboards.

The honest synthesis: keep the manual review. Practitioners are near-unanimous that it is irreplaceable, and Anthropic's guidance to read sampled transcripts weekly says the same thing from the other direction. Automated evals are the regression net that lets you ship; human reading is how you find out what to put in the net.

These come from real vendor evaluations and real code reviews. Each one is checkable in a single question.

The Frenchy Digital red-flag list for agent evaluation claims and eval harness design, 2026.

Ask any agent vendor for one artifact: a sample of failed traces with the grading rationale attached. A vendor that has evaluation can produce it in an afternoon. A vendor that has a benchmark score cannot produce it at all.

— Frenchy Digital buyer’s principle

These are the bands Frenchy Digital uses to scope agent engagements in 2026. Evaluation and observability are inside the scope of every band rather than a later phase, because retrofitting them means you have no history for the period anyone will eventually ask about.

Frenchy Digital cost bands for AI agent engagements, 2026.

Senior-led delivery runs $150 to $225 per hour, and ongoing retainers run $2,500 to $9,500 per month covering model and dependency upgrades, eval set expansion from production incidents, judge recalibration, incident response, and a quarterly technical review. Every engagement carries a 30-day post-launch warranty, and you receive a written scope with a fixed-price phased proposal within 5 business days of the discovery call.

One budgeting note. The evaluation harness is largely a fixed cost, paid once and reused by every subsequent workflow. The first agent pays for the harness, the tracing, the judge calibration process and the CI wiring. The fourth agent inherits all of it and adds only its own cases. Teams that sequence their agents get substantially better economics than teams that run four pilots in parallel, each with its own ad-hoc evaluation and none of them comparable.

Everything above improves your evidence. None of it produces certainty, and several parts of the practice are less established than the confident tone of most writing on the subject suggests. Here is what we tell clients.

Honest limitations of agent evaluation practice as of August 2026.

Two of those rows deserve elaboration because they are the ones most often glossed over.

Model-version drift is real and badly under-measured. The canonical citation remains Chen, Zaharia and Zou, who found GPT-4's prime-identification accuracy fell from 84% to 51% between March and June 2023. That result is nearly three years old and we found no 2026 replication of it. So the honest statement is: this happened once, was rigorously documented, and nobody has systematically re-checked it since. That is a reason to pin versions and re-run your suite on every change, not a reason to claim a current drift rate you cannot source.

Agent-specific rollout practice is not documented anywhere credible. The shadow-to-canary-to-percentage-rollout framing appears constantly in agent content, and we could not trace it to a single primary engineering write-up about agents specifically. It comes from ordinary progressive delivery, where it is well established. Borrow it — it is sound — but be honest with your team that you are extrapolating from web-service practice rather than following a published agent methodology, because the failure modes are not identical. A canary that measures error rate will not catch an agent that succeeds while doing the wrong thing.

Finally, the boundary that holds across this entire cluster: none of this makes an agent safe. Evaluation reduces uncertainty about behaviour you thought to test. Prompt injection remains unsolved, and your eval harness — particularly any LLM judge inside it — is one more surface for it. The correct posture is defense in depth and blast-radius reduction, and it applies to the measurement pipeline as much as to the product.

None of this argues against building. It argues for building the measurement alongside the agent, choosing one workflow with a defensible baseline, and being honest inside your own organization about which numbers are evidence and which are marketing. Teams that get value from agents are the ones that instrumented the before-state.