LLM Evals: How Do You Test an AI Feature Before It Ships?

Thirty-one percent of teams running LLM applications don't evaluate output quality at all, and only 17% have automated it (DataTalks.Club AI Engineering Survey 2025–2026). So when someone edits a prompt and ships it, there's usually nothing standing between that change and your users. The edit either helps or quietly breaks something, and nobody knows which until a complaint arrives.

Evals are the thing that's supposed to stand there. If observability tells you how a feature behaves after it ships, an eval is the gate before it ships — the LLM equivalent of a test suite you run in CI. This post covers what an eval actually is, what to test, whether you can trust an LLM to grade itself, and how to tie the whole thing to the prompt version that triggered it.

Key Takeaways

• 31% of teams run no LLM evaluation and only 17% automate it (DataTalks.Club, 2025–2026) — most prompt changes ship ungated.
• An eval is a test for non-deterministic output: you score quality against a fixed dataset instead of asserting an exact string.
• LLM-as-a-judge correlates with human scores but isn't a free pass — one 2025 study measured 81.32% and 68.51% correlation depending on the task (ISSTA 2025).
• Evals only protect you when each run is tied to a specific prompt version, so a regression points at a diffable change.

What is an LLM eval?

An LLM eval is a repeatable test that scores model output for quality against a fixed set of inputs, rather than asserting one exact correct answer. You collect a dataset of representative cases, run your prompt over them, and grade the results — by exact rules, by an LLM judge, or by a human. The score is the thing you gate on.

It exists because the usual test doesn't fit. A unit test asserts add(2, 2) === 4 and fails loudly when it isn't. An LLM call rarely returns the same string twice, so output === "expected" is useless. You're not checking for an exact match; you're checking whether the answer is correct, on-format, and safe across a spread of inputs. The output is probabilistic, so the test has to be too.

That shift — from "is it exactly right?" to "how often is it good enough?" — is the whole mental move. Get it, and evals stop feeling like a research exercise and start feeling like CI.

Why do most teams skip evals?

Most teams skip evals because the feature looks like it works in the demo, and writing a good test set is more effort than tweaking a prompt until the one example you're staring at looks fine. The same survey that found 31% evaluate nothing puts automated evaluation at just 17% (DataTalks.Club, 2025–2026). The rest are eyeballing outputs and shipping on vibes.

The cost of that lands later, as the quiet failure mode. In Stack Overflow's 2025 survey of roughly 49,000 developers, 66% named "AI solutions that are almost right, but not quite" their single biggest frustration, and 45.2% said debugging AI-generated output takes longer than expected (Stack Overflow, 2025). "Almost right" is exactly what a single hand-checked example will never catch — you tuned the prompt to that case, so of course it passes.

Our take: The reason eyeballing feels sufficient is that it always works on the example in front of you. An eval set's only job is to hold the cases you aren't looking at — the edge inputs, the formats that broke last month, the one phrasing that makes the model refuse. That's where regressions actually live.

This isn't a fringe concern, either. Evaluation is consistently named the top unsolved problem in a16z's 2025 survey of 100 enterprise CIOs, ahead of both cost and latency, with internal benchmarks and golden datasets cited as the first filter enterprises reach for (a16z, 2025). The teams furthest along treat their eval set as core infrastructure, not a nice-to-have.

What should you actually test?

Test four things: capability, regression, safety, and format. Each answers a different question, and the cheapest way to cover them is a single "golden dataset" — a curated set of input/expected-behavior pairs you grow over time, especially every time a bug slips through.

Capability evals are where you start, but regression evals are where the payoff compounds — every production bug becomes a permanent test case, so the same mistake can't ship twice. Format evals matter more than people expect: if a downstream system parses the output, a drift from valid JSON is a hard outage, not a soft quality dip. Pairing evals with structured outputs turns most format failures into something you can assert mechanically, no judge required.

Start small. Twenty to fifty well-chosen cases that cover your real input distribution beat a thousand synthetic ones that all look alike.

Should you trust LLM-as-a-judge?

Use it, but calibrate it — don't assume it's right. LLM-as-a-judge (one model scoring another's output) is the only way to grade open-ended quality at scale, and it correlates with human judgment well enough to be useful. A 2025 study measured Pearson correlations of 81.32% on code translation and 68.51% on code generation between an LLM judge and human scores (ISSTA 2025). Good, but not gospel — and notice how much it swings by task.

So judges aren't a free pass; they're a tool with known blind spots. They drift, they favor longer answers, and their agreement with humans varies by domain, which means you have to validate the judge against a human-labeled sample before you trust its scores. The practical recipe is a layered one:

Assert mechanically wherever you can, judge with an LLM where you can't, and keep a small human-labeled set to keep the judge honest. A judge you've never checked against a human is just another unverified model in your pipeline.

How do evals fit with versioning and observability?

Evals only protect you when each run is pinned to the exact prompt version that produced it. Run your eval suite against a prompt change before you merge; if the score drops, you block the change the same way a failing test blocks a deploy. That loop only works if "the prompt" is a specific, addressable version — not a string someone edited in passing.

That's the catch for most teams. When a prompt is hardcoded in application code, an eval run can't cleanly say "version 7 scored 92%, version 8 scored 78%," because there is no version 8 — just a diff buried in an unrelated deploy. Tie evals to a real prompt version history and the picture sharpens: every eval result maps to a diffable change, a regression names its own cause, and a rollback is one click instead of another release.

The three layers are one system. Versioning gives you something to point at, evals gate it before ship, and observability watches it after — and all three break the moment prompts stop being first-class artifacts. That's also why managing prompts as a team and evaluating them well turn out to be the same discipline. PromptVault gives every prompt version history, diffing, and instant rollback, so an eval score ties back to an exact change you can review and revert.

Frequently Asked Questions

What's the difference between evals and observability?

Evals measure quality against a fixed test set before you ship — a gate. Observability watches what actually happens after you ship — a feedback loop. You need both: evals catch known regressions pre-deploy, observability catches the unknown ones in production. With 31% of teams evaluating nothing (DataTalks.Club, 2025–2026), most are missing the gate entirely.

How many test cases do I need to start?

Fewer than you'd think — 20 to 50 cases that reflect your real input distribution beat a thousand near-identical synthetic ones. Coverage of edge cases matters more than raw count. Then grow the set every time a bug reaches production, since 66% of developers cite "almost right" outputs as their top frustration (Stack Overflow, 2025) — each one is a missing test case.

Can I trust an LLM to grade its own output?

With calibration, yes. LLM-as-a-judge correlated with human scores at 81.32% and 68.51% across two tasks in a 2025 study (ISSTA 2025) — useful, but it varies by domain and drifts over time. Validate the judge against a small human-labeled sample first, and assert mechanically wherever the output is structured enough to check by rule.

Do small teams really need evals?

Yes, arguably more, because a small team feels every regression directly. You don't need a platform to start — a script that runs your prompt over 30 saved cases and reports a pass rate covers most of the value. Given evaluation is the top unsolved enterprise AI challenge even at scale (a16z, 2025), starting early is a real advantage.

The takeaway

An eval isn't a research project you defer until you have time. It's the test suite that stands between a careless prompt edit and your users — and right now, for nearly a third of teams, nothing stands there at all. The numbers are blunt: most teams evaluate nothing, the dominant failure is the quiet "almost right" kind, and the practitioners furthest ahead treat their golden dataset as core infrastructure.

Start with 30 cases and a pass rate. Add a regression case every time something breaks. Calibrate your judge against a few human labels. And tie every run to a prompt version, so a dropped score points at a change you can actually diff and revert — which is exactly what PromptVault is built for. For the foundation underneath all of it, start with why prompts are config, not code.