Evaluation and Benchmarks

The right question is not “which model is best” but “which model is best for this workload, against my constraints, with my failure tolerance.” That answer never lives on a leaderboard.

SWE-bench Verified measures real-world software engineering: 500 GitHub issues from 12 Python repositories, each manually reviewed by 93 contracted developers (OpenAI, August 2024). It dominated coding evaluation through 2025. Then in February 2026, OpenAI deprecated it — frontier models could reproduce verbatim gold patches on certain instances, which is a contamination signature, not a capability signal (OpenAI, February 2026). SWE-bench Pro, with a standardised scaffold and multi-language coverage, is the successor; current frontier sits in the mid-40s, well below saturation.

GPQA Diamond is the hardest 198 questions of the Graduate-level Physics, Chemistry, and Biology benchmark. PhDs in the question’s own field score about 65%; non-experts with full web access manage 34%. Frontier models now sit above 94% — Claude Opus 4.7 at 94.2%, Gemini 3.1 Pro Preview at 94.1%, GPT-5.4 at 92.0% as of May 2026 (Artificial Analysis). The gap between frontier models and domain PhDs has widened from +7 points in late 2024 to +24 points in early 2026. Saturation is imminent.

AIME 2025 uses the 30 problems from the 2025 American Invitational Mathematics Examination. The top models cluster between 95% and 96% — Kimi K2.5, GLM-4.7, GPT-5.2 (xhigh) all within half a point. When the top three are tied to within rounding error, the benchmark is no longer ranking them; it is measuring noise.

FrontierMath is the exemplar of “still has headroom.” Epoch AI commissioned 350 original research-level math problems from mathematicians including IMO gold medalists and Fields medalists. Each problem requires hours to days of work from a domain researcher. The top score as of April 2026 is GPT-5.5 Pro at 52.4%. At that level, the benchmark still discriminates — and likely will for a year or two.

ARC-AGI-2 tests fluid reasoning on novel visual puzzles. Frontier models broke through the 85% grand prize threshold in early 2026 — GPT-5.5 at 85%, GPT-5.4 Pro at 83.3%, Gemini 3.1 Pro at 77.1% (ARC Prize). The Prize team responded by releasing ARC-AGI-3, on which the same frontier models score below 1%. The frontier of generalisation got reset.

Chatbot Arena — rebranded LMArena in January 2026 — is the only major eval driven by human preference rather than ground-truth labels. Users pick between two anonymised responses; the system fits a Bradley-Terry rating from the pairwise comparisons (Chiang et al., ICML 2024). The catch: a 2025 Cohere/Princeton analysis found that Meta, OpenAI, Google, and Amazon had been submitting many private model variants and publishing only the highest-scoring — a selection effect worth up to 100 Elo points of inflation. Crowdsourced preference is real signal, but it has the same Goodhart problem as every other ranking once labs optimise for it.

The pattern is unmistakable. Every benchmark that gains industry traction enters a saturation curve. Two-year-old benchmarks are historical context. Six-month-old benchmarks are the ones being optimised for right now. Anything in production should treat benchmark scores as a coarse capability filter — not a substitute for evaluation on your actual workload.

The three biases

Position bias. When given two responses to compare, judges prefer the response that appears first (or sometimes second — it varies by judge family). The effect is strongly modulated by the quality gap: when the two responses are close, position bias dominates (Shi et al., IJCNLP 2025). Mitigation: present each pair in both orderings and average the results.

Verbosity bias. Judges prefer longer responses, even when the additional length adds no informational content. This is an artifact of generative pre-training and RLHF — the same training process that makes models hedge and pad in their own outputs also biases them to reward those patterns in others (Justice or Prejudice?, 2024).

Self-preference. Judges prefer outputs that look like their own. Wataoka et al. (2024) traced the mechanism: judges assign higher scores to outputs with lower perplexity, regardless of authorship. A model’s own outputs naturally have low perplexity under its own distribution — so it preferentially rewards them. Mitigation: use a judge from a different model family than the candidate.

A worked rubric: RAG faithfulness

Let’s make this concrete with the eval most production teams encounter first — whether a RAG system is staying grounded in its retrieved context.

The metric. Faithfulness measures how factually consistent a generated answer is with the documents the retriever returned. It catches a specific failure — the model adds claims that are not supported by the context. The Ragas formulation: faithfulness = (claims supported by the context) / (total claims in the response). Range 0 to 1. Higher is better (Ragas docs).

Why this rubric works. It atomises the judgment. Instead of asking the judge “is this answer faithful, on a scale of 1–5” — which invites verbosity bias and rubric drift — it asks “list the claims, check each one, return a ratio.” The output is structured. The score is computed, not chosen. The evidence quote is mandatory, which gives you an audit trail when you need to debug a low score.

A concrete example. Question: which compression algorithm does MongoDB Atlas use for collection data? Retrieved context says: “Atlas uses Snappy by default; you can switch to zstd or zlib at collection creation time.” The generated answer claims Atlas supports Snappy, zstd, zlib, and LZ4. The judge extracts three claims: (1) Snappy default — supported; (2) zstd and zlib — supported; (3) LZ4 — absent from context (and incidentally false). Faithfulness = 2/3 = 0.67. The audit trail tells you exactly which claim was unsupported, so you know whether to fix the retriever (missing context) or the prompt (model hallucinated).

That structure — atomic claims, per-claim verdicts, computed score — is what separates an evaluation from a vibe check. Apply the same pattern to other rubrics: code correctness becomes “list the test cases, check each one.” Answer relevance becomes “list the question’s sub-asks, check coverage of each.” The more atomic the judgment, the less room for bias.

When not to use LLM-as-judge

It is not a universal hammer. The judge needs the capability to grade the task. A judge weaker than the candidate is unreliable, especially on tasks the judge cannot itself solve (advanced math, complex multi-step reasoning, domain-specific code review). For these, you need a stronger judge model, ground- truth labels, or human review on a sampled tier.

LLM-as-judge also costs real money at scale. A faithfulness eval over 10K traces with a GPT-5-class judge is non-trivial in dollar terms. The usual production setup: cheap automatic checks (schema validity, exact match, regex) catch the obvious failures, LLM-as-judge handles the nuanced metrics, and a sampled human review tier calibrates the judge itself.

Start from production traces, not synthetic prompts. Sample 50 to 200 real user inputs from your application. Synthetic prompts written by your team will not match the distribution your users actually generate — they will be cleaner, better-formed, and miss the long tail of real-world phrasing.

Annotate ideal outputs. Have a domain expert write what the correct answer looks like for each input. This is the only step that cannot be automated. The eval set is only as good as these annotations — shortcuts here poison every downstream metric.

Layer the graders. Three tiers, applied in order: automatic deterministic checks (exact match, JSON schema validity, regex, latency, token count, cost) catch the largest class of failures cheaply; LLM-as-judge handles faithfulness, answer relevance, tone, rubric scoring; a rotating 5–10% human sample calibrates the LLM judge and surfaces failures both other graders miss.

Version everything. Prompt version, model version, retrieval config, eval set version, judge model and judge prompt. When a metric regresses, you need to know what changed.

Wire it into CI. Eval runs on every prompt change, every retrieval config change, every model swap. If it does not run automatically, it does not run. The cost of running a 200-example eval suite per change is dollars; the cost of shipping a regression to production is days of debugging plus user trust.

Iterate via the flywheel. Every production failure — bug report, support ticket, thumbs-down — becomes a new eval case. The eval set grows monotonically. Over time, it becomes a high-fidelity map of every failure mode your system has ever hit, and the regression suite catches them all before they ship again.

The eval set is your single most valuable artifact. It captures what “good” means for your product in a form that survives model upgrades, prompt rewrites, and team changes. Treat it the same way you treat your production code — version it, review it, protect it.

A brief word on safety

Safety evaluation is the same discipline applied to adversarial inputs and undesired outputs. The pattern does not change — you collect inputs (jailbreak prompts, prompt injection attempts, harmful requests), define failure modes, and run automated checks in CI. What is different is the input distribution. Adversarial inputs are not in your production traces; you have to either curate them yourself, use a published red-team suite, or generate them programmatically. The UK AI Security Institute’s Inspect framework is the most credible open-source toolkit for this — it includes a sandboxing layer for agent tool calls, which matters once your system can execute code or hit APIs. The recurring finding across published red-team studies is that role-play jailbreaks (fictional-character framing, hypothetical scenarios) succeed against frontier models more often than direct attacks. Treat safety the way you treat any other failure mode: a defined set of inputs, a defined set of bad outputs, automated checks on every change. The moment safety eval lives in a Word document instead of CI, it is decoration.

The recurring production pattern: a lightweight framework for CI/CD gating (Promptfoo, or Ragas for RAG) paired with an observability platform for traces and human annotation (Braintrust or LangSmith), plus Inspect for safety and agent work. There is no single tool that does all three jobs well, and trying to force it usually means missing one of them entirely.