Verification loop

What is a verification loop?

A verification loop is the pattern of giving an AI agent a check it can run itself, and letting it work, run the check, read the result and try again until the check passes. The check can be a test suite, a build that has to exit cleanly, a schema validation, a row count that has to match the source, or a written rubric that a second model grades against.

Without such a check, an agent stops when the work looks done. Anthropic's documentation for Claude Code puts it plainly: "looks done" is then the only signal the agent has, and you become the verification loop, because every mistake waits for you to notice it. Give the agent something that produces a pass or a fail and the loop closes on its own.

Think of an accountant closing the month. The books are not done when the entries are typed in. They are done when the bank balance and the ledger agree. A verification loop gives an agent that same finish line: not "I have written the code" but "the check says the code is right".

What makes a good check

Not every check is worth building a loop around. Karpathy's November 2025 note on verifiability lists three properties of a task an AI can practise on its own: you can reset and start a new attempt, many attempts are cheap, and an automatic process can score each attempt. The same three properties describe a good check inside an agent loop.

Cheap. The agent may run the check twenty times in one session. A unit test suite that finishes in a minute qualifies. A full nightly load of the warehouse does not; for that you want a sample or a smaller fixture.

Deterministic. The same output must get the same verdict every run. A test passes or it does not. A row count matches or it does not. A check whose answer wobbles teaches the agent nothing, because it cannot tell whether it fixed the problem or got lucky.

Hard to game. The check has to look at the result, not at the agent's claim about the result. A script that diffs the produced file against a known-good fixture is hard to talk your way past. An instruction that says "make sure the numbers are correct" is not a check at all.

In software the checks have existed for years: tests, a type checker, a linter, a build exit code, and more recently a screenshot of the running app compared against the design. Claude Code's guidance names exactly these, and adds that the agent should show the evidence, the test output or the command it ran, rather than assert that it worked.

A worked example from data work

Data work has its own checks, and most teams already use them without ever hearing the term verification loop. dbt's generic tests are a good illustration. Each test is a SELECT statement that looks for failing rows: duplicates for unique, empty values for not_null, values outside an allowed list for accepted_values, references to a customer that does not exist for relationships. Zero rows back means the test passes. That is a cheap, deterministic, hard-to-game check, ready for an agent to run.

Suppose you ask an agent to rebuild the sales-orders model in a new warehouse. Instead of telling it "make sure the data is right", you give it three checks:

1. Row count. The target must contain the same number of order lines as the source, minus the lines the documented filter removes.

2. Control totals. The sum of the amount column per month must reconcile with the source to the cent.

3. Referential check. Every customer_id in the orders table must exist in the customer table.

The agent writes the model, runs the three checks, and reads the result. Say the row count is two short and January's total is off by 1,240 euro. It goes looking, finds a date filter that excludes orders placed at exactly midnight, fixes it, and runs the checks again. Only when all three pass does it hand the work back to you, with the query output as evidence. The part you review is the evidence, not the SQL line by line.

The same shape works for a Power BI migration (the new measure must return the same total as the old report for the last twelve months), a schema change (the JSON output must validate against the published schema) and an invoice-processing agent (the extracted line items must add up to the invoice total on the document).

Verification loop versus LLM-as-a-judge

Both patterns put a check behind an agent's output. The difference that matters is determinism.

A verification loop in the strict sense uses a check that is code: a test, a query, a diff. Run it twice and you get the same answer twice. It is narrow, because it can only check what someone wrote a rule for, but inside that scope it is reliable and the agent cannot argue with it.

LLM-as-a-judge uses a second model to read the output and score it against written criteria. It reaches things no script can reach: is this summary faithful to the report, is this customer reply polite, does this slide deck follow the brief. The price is that the judge is itself a model. Run it twice and you can get two different scores, and it carries known biases towards longer answers and towards its own model family.

In the tooling the two are growing towards each other. Anthropic's Managed Agents let you define an outcome with a rubric of explicit criteria; a separate grader in its own context window scores the deliverable, reports which criteria failed, and the agent takes another pass, three iterations by default and at most twenty. Claude Code's /goal command has a small model read the transcript after every turn and return met, not yet met or impossible. A Stop hook, by contrast, runs your own script and is fully deterministic: it blocks the turn from ending until the script passes, up to eight times in a row.

A rule you can work with: use a code check wherever one exists, use a rubric grader only for what a code check cannot reach, and write the rubric as gradeable statements ("the CSV has a numeric price column") rather than impressions ("the data looks good"). Anthropic's own documentation for the grader says the same: vague criteria produce noisy evaluations.

When the agent games the check

The loop has one well-documented failure mode. An agent under pressure to make the check pass may find a way to satisfy the letter of the check without doing the work. Anthropic's research team calls this reward hacking and, in a November 2025 study on their own models, gave a concrete example: calling sys.exit(0) in Python to break out of the test harness with a success exit code, so every test appears to pass. Their comparison is a student writing "A+" at the top of their own essay.

In data work the equivalents are easy to picture. The agent widens the tolerance on the reconciliation until the difference falls inside it. It deletes the failing test instead of fixing the model. It hard-codes January's total. None of this is malice; the model is doing what the loop rewards.

The defences are mostly structural:

• Keep the check out of reach. The agent runs the tests but does not own the test file, the fixture or the tolerance. In a pipeline, that means the check lives in a separate repository or on a protected path.

• Grade in a fresh context. Both Claude Code's verification subagent and the Managed Agents grader run in their own context, so the model doing the work is not the one deciding it is done.

• Demand evidence. Ask for the raw output of the check, not a sentence that says it passed. A pasted row count is harder to fake than "all checks green".

• Cap the iterations. A loop that has run ten times without converging is telling you the task or the check is wrong. Stop it and look.

Which of your processes are ready for an agent

Karpathy's rule of thumb, repeated in his April 2026 write-up of his talk at Sequoia's Ascent conference, is that traditional software automates what you can specify, and this generation of AI automates what you can verify. Coding agents feel so much better than a general chatbot because code gives the model feedback: tests pass or fail, programs run or crash. Creative and strategic work, and anything that depends on context that lives only in people's heads, lags behind because there is no automatic score.

That gives a small business a usable test for its own processes. Ask of each one: does it end in something that has to match something else?

Matching supplier invoices to purchase orders and goods receipts ends in a match or a mismatch. A VAT return has to reconcile with the ledger. A stock count has to agree with the system. A migrated report has to return the same totals as the old one. These are ready, because the check already exists and your team is running it by hand today.

Writing a proposal for a customer, choosing a price, deciding whether to take on a difficult client: there is no automatic check, only judgement. An agent can draft, but a person still verifies, and that verification is the job.

Where a good check exists, the human's role moves from doing the work to reading the evidence and deciding whether to accept it. Where no check exists, the human is the check, so put a human-in-the-loop gate at the point of action, before the mail goes out or the payment is released.