AI benchmarks

What are AI benchmarks?

An AI benchmark is a fixed set of tasks with a fixed way of scoring them, published so that anyone can run any model against it and compare the results. A leaderboard is the ranked list of those results. When a lab announces a new model with a chart full of bars, each bar is a score on one benchmark.

Benchmarks exist because the field needed a shared yardstick. Two people saying "this model feels smarter" is not a comparison. The same 500 coding tasks, run under the same rules, give a number a competitor can reproduce next month. That is the whole value: a benchmark is a common exam, not a description of your work.

How a benchmark is built and scored

Every benchmark has three parts: a set of tasks, a definition of a correct answer, and a rule that turns the results into one number. What differs is where the tasks come from and who decides what counts as correct.

Take a coding benchmark built from real repositories. A team at Princeton collected about 2,300 real bug reports from twelve open source Python projects, together with the fix a human developer eventually merged. The model gets the codebase and the bug report, writes a patch, and the patch is judged by running the project's own tests: the ones that failed before must pass, the ones that passed must still pass. No human opinion is involved. The score is the share of bugs fixed. When this benchmark came out in late 2023, the best model fixed under 2 percent.

That is the strongest kind of benchmark, because the answer key is executable. The weakest kind is a multiple-choice set, where the answer key is a letter and a model can score by pattern matching. In between sit benchmarks judged by people, or by a language model used as a judge.

Four benchmarks you will see on launch charts

SWE-bench Verified is the coding benchmark above, cut down to 500 tasks that human annotators checked for being clear and fairly tested.

OSWorld measures computer use. The model gets a real desktop with real applications, a spreadsheet, a browser, a file manager, and a task such as changing a setting or moving data between two programs. A script checks the end state of the machine. It has 369 tasks. People succeed on about 72 percent of them. The first models managed about 12 percent, and Stanford's 2026 AI Index reports the best score now at about 66 percent, within six points of the human number.

Humanity's Last Exam is an exam-style set of 2,500 questions written by close to a thousand academics and researchers at over 500 institutions, chosen to be hard for models. It was built because models were scoring above 90 percent on the older knowledge tests, which made those useless for telling models apart.

Arena leaderboards work differently. On a site such as LMArena you type a question, two anonymous models answer, you vote for the better one, and only then are the names shown. The votes are combined into a rating that works like a chess ranking. This is the closest thing to measuring what people prefer, and it inherits every bias people have.

Why launch charts are walls of bars

A launch chart shows eight or twelve benchmarks because no single one covers what a model does, and because every lab picks the set where its model looks good. The chart is drawn by the party being measured, usually from runs the lab did itself, with its own prompting and its own number of attempts. Epoch AI, an independent research group, keeps a database that separates results it ran itself, results run by the benchmark's makers, and results reported by the model developer. Launch charts are made of the third kind, and an independent rerun does not always land on the same number.

The other reason the charts keep growing is that the benchmarks on them keep dying. The AI Index puts it plainly: evaluations intended to be challenging for years are saturated in months. On Humanity's Last Exam, frontier models gained 30 percentage points in a single year. Once every leading model scores in the nineties, a benchmark stops telling anyone anything, and a new one appears on the next chart.

What to watch out for with AI benchmarks

Contamination. Benchmark tasks are public and models are trained on the public internet. If the test questions and answers end up in the training data, the model has seen the exam. Terminal-Bench, a benchmark of command-line tasks for agents, carries a line in capitals on its site asking that its data never appear in training corpora, which tells you how real the problem is. Contamination cannot be ruled out from the outside.

Saturation and flawed questions. A benchmark where the top models all score 95 percent no longer separates them, and part of what is left may be bad questions. A review cited by the AI Index found the share of invalid questions in popular benchmarks ranged from about 2 percent to over 40 percent depending on the set. A model can lose points for being right.

Gaming the leaderboard. In 2025 a team led by Cohere Labs, with researchers from Princeton, Stanford and MIT among the authors, studied how the arena was being used. Labs could test many private versions of a model, see the scores, and publish only the best. Meta tested 27 private variants in the month before its Llama 4 release. In the team's simulations, submitting 20 variants and keeping the best added roughly 50 rating points, twice the whole gap between first and sixth place in the AI Index's March 2026 snapshot. The same paper found that two labs each received about a fifth of all arena conversations to learn from, while dozens of open models together got under a tenth. Training on that data lifts your arena score without improving your model. The AI Index summarises it carefully: arena standing may partly reflect adaptation to the platform rather than general capability.

Voting bias. The arena's own analysis found that answer length is the single biggest style factor in who wins a vote, with headers, bold text and lists behind it. Since 2024 the leaderboard offers a corrected view, and some models move five places when it is switched on. A model tuned to please voters climbs the chart; the sycophancy entry explains where that leads.

Narrow tasks. Every benchmark measures one slice. A model can lead on coding and be ordinary at reading a Dutch contract. The jagged intelligence entry covers why a high score on a hard task says so little about an easy one next to it.

Public benchmark versus your own eval set

The cleanest way to see the difference is to ask what question each one answers.

A public benchmark answers: how does this model compare to other models on a fixed, public task, under the rules of that task? It is a ranking tool. It can tell you that model A is generally stronger at code than model B, and that the gap between the top three is small.

Your own eval set answers: does this model do my task well enough to trust it? It is a go or no-go tool. It holds your invoices, your customer emails, your product names, your Flemish, and a definition of correct that your team wrote. Nobody outside your company can run it, which is exactly why nobody could have trained on it.

A model can win the first question and fail the second. It can also lose the first by a few points and still be the best choice for you, because it is cheaper, faster, or good at the one thing you need. No benchmark score says anything about your invoices. Only a test on your invoices does.

How to use benchmarks when you choose a model

1. Shortlist with benchmarks. Look at two or three benchmarks close to your task, from an independent source such as Epoch AI rather than the launch chart, and drop anything far behind. Treat gaps under five points as a tie.

2. Choose with your own evals. Run the shortlist on fifty real cases from your process, each with a known correct answer. The evals entry describes how to build that set. The winner on your cases is the winner, whatever the leaderboard says.

3. Re-run at every release. A new model claims a jump on the chart. Your fifty cases tell you within an hour whether the jump reached your task.

A worked example. You want line items out of supplier invoices, some scanned, some in French. The launch chart shows model A at 74 and model B at 71 on the coding benchmark. That is 15 tasks out of 500 and has nothing to do with invoices, so both make the shortlist. On your 50 invoices, A gets 41 right and B gets 46, and B costs a third less per page. B is your model. The benchmark kept the weak options out of the room; your eval did the choosing.