LLM Evals: How to Build an Evaluation Harness Before You Ship

The Short Answer

An LLM evaluation harness is a versioned test set, a set of scorers, and a CI job that can block a merge. Build it in this order: label 100 real production traces, write code graders for every check that can be expressed as an assertion, add a small human-labeled gold set, calibrate a cheap LLM judge against that set and measure its agreement, then gate on per-category minimums plus a zero-tolerance safety set. Do not start by buying a platform.

Start With 100 Labeled Traces, Not a Platform

If you cannot name the failure you are testing for, you do not have an eval. You have a demo with assertions bolted on.

The highest-return hour here is reading real production traces and labeling each one good or bad. Hamel Husain argues for binary labels over a 1-to-5 scale on operational grounds: granular ratings are more onerous to manage than binary ratings, and a label nobody can apply consistently is not data. A hundred traces shows you the shape of your failures.

What you find is rarely what you expected. A large share of real failures are mechanical: malformed JSON, a missing field, a citation ID absent from the corpus, a tool called with the wrong argument type, a blown latency budget. Every one of those is an assertion you can write in code.

The eval sets that never move are the ones written by the engineer who wrote the prompt, on the same afternoon, from memory. They encode the happy path, pass at 98 percent forever, and support tickets climb anyway.

Code Graders First, Judges Only for What Is Left

If a check can be expressed as code, never spend a judge call on it. Code graders are deterministic, free, and immune to model drift. OpenAI's Graders API defines four grader types: string check, text similarity, score model, and python. The useful question about any of them is whether it needs a model at all.

Write the numeric threshold for every scorer before the suite runs: the minimum exact-match rate, the minimum similarity score, the maximum p95 latency, the maximum cost per request. Picking a threshold after seeing the score is how teams talk themselves into shipping a regression.

Your Judge Is a Model in Production and Needs Its Own Eval

The pitch for LLM-as-a-judge is that it replaces vibes with numbers. What it actually gives you is a noisy estimator with documented directional biases, which is a different thing.

G-Eval is one of the most cited judge methods in the literature, and its authors report a Spearman correlation of 0.514 with human judgments on summarization, which the paper describes as outperforming all previous methods by a large margin. That is the published state of the art. Wiring a 0.5-correlation instrument directly to a merge gate produces confident wrong decisions at scale.

These biases have direction, so more samples do not average them away. Judging the Judges, a position-bias study covering 15 judges, 22 tasks, and roughly 150,000 evaluation instances, found that the bias varies by judge and task and is only weakly influenced by prompt length. What drives it is the quality gap between the two candidates, which means the bias is worst when the candidates are close, and that is exactly when you are asking the judge to decide something. Self-Preference Bias in LLM-as-a-Judge found that judges score lower-perplexity text higher than human evaluators do, whether or not the judge generated it. Judges over-reward writing that sounds familiar and confident, which is the failure mode that produces fabricated policies stated with total certainty.

Four rules follow, all cheap to implement:

1. Measure judge agreement against human labels on a held-out set before the judge gates anything, and re-measure whenever you change the judge model, the judge prompt, or the rubric.
2. Pin the judge model version in the eval config, separately from the model under test. If both float, a score change is unattributable.
3. In pairwise comparisons, score every pair in both orders and average.
4. Do not use the same model family as generator and judge for anything that gates a release. Anthropic's eval documentation gives the same advice on using a different model for grading.

How Many Test Cases Do You Actually Need?

More than you have. Evan Miller's Adding Error Bars to Evals, written at Anthropic, runs the power analysis: detecting an absolute difference of 3 percent between two models at 80 percent power and 5 percent significance needs about 969 independent questions. The typical suite is 40 rows in a spreadsheet, and the typical decision is a 5-point swing sitting inside the noise.

The same paper puts clustered standard errors on the DROP benchmark at 3.05 times the naive ones, because questions drawn from the same source passage are not independent samples. If your items come in groups, several questions per document or several turns per conversation, compute clustered standard errors or you will ship noise as a win and the same noise back as a regression.

Two moves make a small suite usable. Run both variants on the identical item set and do inference on per-item paired differences rather than on the two aggregate scores. Then resample: Miller shows that going from one run per question to two cuts variance by a third.

Volume beats polish. Anthropic's eval design guidance puts it plainly: more questions with slightly lower signal automated grading beats fewer questions with high-quality hand grading. Its worked examples run to 1,000 tweets for a sentiment eval and 500 simulated patient queries for a PHI leakage eval.

How Do You Evaluate an Agent?

Grade the end state, not the transcript. An agent that narrates issuing a refund, cancelling the order, and emailing the customer passes every transcript grader while none of it happened. tau-bench takes the right approach: it compares the database state at the end of the conversation against an annotated goal state.

Then report pass^k alongside pass@1. tau-bench introduced pass^k to measure whether an agent solves the same task on all of k independent trials, and its authors report state-of-the-art function-calling agents succeeding on under 50 percent of tasks, with pass^8 under 25 percent in the retail domain. Run the arithmetic on your own number: a 90 percent single-run success rate compounds to 43 percent across eight runs. Users hit that tail every day.

Task length is what widens the gap. Each additional tool call is another independent chance to fail, and multi-step runs fail at the joins: a stale record ID from step two used in step five, a retry that double-charges. Score the run, and score it repeatedly, because a single green transcript proves almost nothing.

The consequences are on the record. British Columbia's Civil Resolution Tribunal ordered Air Canada in February 2024 to honor a bereavement fare its support chatbot had invented, rejecting the airline's argument that the chatbot was a separate legal entity responsible for its own actions. Cursor's support agent invented a login policy that did not exist in April 2025, users cancelled over it, and the company apologized and refunded them. Replit's coding agent deleted a client's database in July 2025 during a declared code freeze, against explicit instructions. No faithfulness score catches any of that. End-state grading and an adversarial set do.

Keep the Adversarial Set Separate

Quality evals and safety evals fail differently and should not share a pass threshold. Run a separate set covering, at minimum, prompt injection, sensitive information disclosure, and excessive agency, which are LLM01, LLM02, and LLM03 in the OWASP GenAI Security Project's 2026 Top 10 for LLM Applications, published on August 4, 2026. Excessive agency moved up from LLM06 in the 2025 edition, which tells you where the industry thinks the risk now sits.

Gate on per-category minimums plus zero tolerance. One PII leak, one unauthorized tool call, or one fabricated policy statement blocks the release regardless of the aggregate. Aggregates hide category collapse. Take a shape that shows up constantly: a category worth 2 percent of the set falls from 88 percent to 40 percent, the aggregate moves about one point, and the dashboard still reads 91 percent while the highest-value refunds are broken.

Build or Buy, and What It Actually Costs

Start with an open-source harness, a results table in the Postgres you already run, and a CI job. That covers gating, which is the use case that matters first. Inspect, the MIT-licensed evaluation framework built by the UK AI Security Institute, ships with a collection of over 200 pre-built evaluations you can run against any model.

Choose on who does the annotating. Running the job is the easy part, and cost is no reason to skip any of this. Claude Haiku 4.5 lists at 1 dollar per million input tokens and 5 per million output, Gemini 2.5 Flash-Lite at 0.10 and 0.40, both with a 50 percent batch discount, so a nightly judged run over 1,000 cases costs a few dollars.

One constraint overrides the table: do not build on a hosted product you cannot export from. OpenAI's own Evals platform goes read-only for existing users on October 31, 2026 and is scheduled to shut down on November 30, 2026, and Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI. Keep the dataset and the scorers in your repo and treat the platform as a viewer.

Wire It to CI and Treat Model Retirement as a Scheduled Event

A harness in a notebook on one laptop answers no useful question. It has to run on every change, persist results with a timestamp and a model version, and be able to fail a build.

Migration is the case that pays for the effort. Hosted models change under you: Chen, Zaharia, and Zou measured GPT-4 identifying prime versus composite numbers at 84 percent accuracy in March 2023 and 51 percent on the same questions in June 2023. Retirements are routine and dated. Anthropic gives at least 60 days of notice before retiring a publicly released model, and Claude Sonnet 3.7 and Haiku 3.5 retired on February 19, 2026, Sonnet 4 and Opus 4 on June 15, 2026, and Opus 4.1 on August 5, 2026. Requests to a retired model fail outright. The day the deprecation notice arrives is the day you run the full suite against the replacement.

The Pre-Ship Checklist

• 100 real traces read and labeled, failure categories named
• Code graders for every check expressible as an assertion
• Human-labeled gold set with measured judge agreement
• Judge model version pinned separately from the model under test
• Pairwise scoring run in both orders
• Per-category minimums plus a zero-tolerance safety set
• Agents graded on end state, reported as pass^k
• Cost and p95 latency scored in the same run as quality
• Eval inputs excluded from few-shot examples, fine-tuning data, and the RAG index
• Every production incident added as a case the same day, using the exact input
• The suite runs in CI and can block a merge, with an owner named

Scorers that need no reference answer, such as schema validity, PII checks, and latency, can also run as monitors on live traffic. Scorers that compare against a known correct answer stay offline against the fixed dataset.

What This Looks Like in Delivery

Across 7 production AI systems in 5 industries, the harness is the artifact that made the rest defensible. The cybersecurity query engine has run more than 6 months in production with zero bad queries, and that is a claim only an evaluation set makes checkable. The call center QA system reached 100 percent automated coverage where manual review had never exceeded 5 percent, which is worth nothing without scorers you trust. Proof of concept in days, production in 4 to 8 weeks, then 90 days of monitoring, and the suite connects all three phases.

Get a Free Technical Assessment

If you are about to ship an LLM feature and the eval story is a spreadsheet, we offer a free technical assessment: a 30-minute call about your use case, followed by a written roadmap within 48 hours covering eval design, scorer selection, gating thresholds, and CI wiring. There is no obligation, and the roadmap is yours to keep. Book at sasid.ai.