Evals (LLM evaluation)

What are evals?

Evals, short for evaluations, are structured tests for measuring the quality of a language model or an AI application. An eval usually has two parts: test cases that represent the task, and a scoring method that compares the model output with the expected behaviour.

Evals are for AI systems what unit tests and regression tests are for software, with one awkward difference: the answer is often not simply pass or fail. A summary can be partly correct. A chatbot answer can be safe but unhelpful. A retrieval answer can cite the right document and still miss the point.

Without evals, model changes are guesswork. You switch model, rewrite a prompt, change a retrieval setting, or adjust temperature, and someone says it feels better. With evals, you at least have a repeatable signal: what improved, what regressed, what became cheaper, and what became slower.

Types of evals

Multiple-choice benchmarks
MMLU, HellaSwag, TruthfulQA, BIG-bench, and HELM-style benchmark suites are public tests with fixed questions or tasks. MMLU, for example, covers 57 subject areas. These benchmarks are useful for comparing base models, but they are not a substitute for testing your own product.

Reference-based evals
You define an expected answer or expected fields and compare the model output against it. Exact match works for labels, IDs, JSON fields, and classifications. Similarity scores can help for summaries, translations, or search-style answers, though they need human sanity checks.

LLM-as-a-judge
A stronger or separate model scores the output against a rubric. This is useful when exact answers are too narrow: helpfulness, tone, factual grounding, policy compliance, or whether the answer followed instructions. The judge itself needs calibration against human judgement, because it can have systematic preferences.

Human review
People compare outputs, score rubrics, or label failures. Human review is slower and more expensive, but it remains the reference point for tone, usefulness, safety, and domain nuance.

Red-teaming
Red-team evals try to break the system: prompt injection, jailbreaks, unsafe advice, data leakage, hidden assumptions, or harmful edge cases. They are less about a single score and more about finding failure modes before users do.

Public benchmarks versus your own evals

Public benchmarks are useful for orientation. They help you understand whether one model family is generally stronger than another at reasoning, coding, knowledge, maths, or safety. Stanford's HELM work is a good example of a broader benchmark approach that tries to compare models across scenarios and metrics.

But public benchmarks have three limits.

• They are generic. They do not know your support tickets, product names, regional language, data model, or compliance rules.

• They can be contaminated. Popular benchmark questions are public. A model may have seen some of them during training.

• They are format-sensitive. Small prompt and answer-format changes can move scores. Treat benchmark numbers as directional, not as a contract.

Your own evals should sit close to real use. If the application answers customer emails, the test set should contain real or realistic customer emails. If it generates DAX, test against the kinds of models and measures your team actually writes. If it uses RAG, test whether the answer is grounded in the retrieved material, not just whether it sounds confident.

How to build a practical eval suite

1. Collect realistic cases. Start with 20 to 100 examples from logs, support conversations, user interviews, failed outputs, and edge cases. Thirty good cases are better than three hundred vague ones.

2. Define expected behaviour. For each case, write what good means. That may be an exact label, a required JSON shape, a checklist, a refusal rule, or a short rubric.

3. Choose the scoring method. Use exact matching where possible, rule-based checks for format, LLM-as-a-judge for subjective criteria, and human review for the first calibration rounds.

4. Run a baseline. Measure the current prompt, model, retrieval setup, and parameters. That is the reference point for future changes.

5. Run it on every change. Prompt edits, model upgrades, chunking changes, new tools, and parameter changes should all pass the eval suite before release.

Tools help, but the discipline matters more than the brand. OpenAI's current docs describe evaluation best practices and an Evals API, while also noting that the older Evals platform is being deprecated in late 2026. Promptfoo, Langfuse, Braintrust, OpenAI tools, custom scripts, and spreadsheets can all work if the cases and scoring are clear.

What to measure besides quality

Cost
A model that scores three points higher but costs five times more may still be the wrong choice. Track input tokens, output tokens, tool calls, and total cost per successful task.

Latency
A support assistant that answers correctly after twenty seconds may feel broken. Measure p50 and p95 response time, not just average latency.

Stability
Run important evals more than once when generation is stochastic. A system that passes only half the time is not ready for a workflow where consistency matters.

Safety and policy behaviour
Good task performance can hide unsafe edge cases. Keep separate evals for refusal, privacy, prompt injection, and sensitive-domain behaviour.

What to watch out for

Overfitting to the eval
If you tune prompts only until the eval score peaks, the model may get worse outside the test set. Keep a holdout set and refresh examples as real usage changes.

Eval drift
Products, policies, prices, and user language change. A one-year-old eval suite may test yesterday's world. Review the set periodically.

Tiny test sets
Five examples create false confidence. Use a small suite to start, but grow it around real failures.

Judge bias
An LLM judge may prefer longer answers, familiar wording, or a model from the same family. Calibrate it against human labels and keep the rubric specific.

Only testing the model
Most AI failures are system failures: retrieval, tools, permissions, formatting, stale context, bad prompts, or missing guardrails. Evaluate the whole application path, not just the model in isolation.