Autonomous experimentation

What is autonomous experimentation?

Autonomous experimentation is a loop that runs with nobody in the cycle. Something makes a change, measures the result against a score it works out for itself, keeps whatever wins, and starts again. You set it going in the evening and read a ranked list in the morning. Change a setting and measure is what tuning has always been; what is new is that both halves happen without you, hundreds of times over.

Three things have to exist before such a loop can run at all, and most ideas fail on one of them.

1. A change the system can make on its own. A config value, a prompt string, a chunk size. Something that takes effect by writing a value, not by a person rebuilding a pipeline.

2. A score it can compute without asking anyone. How many of 120 stored questions returned the right passage, or how far last year's forecast sat from the actuals. If a person has to judge each result, you have a queue of work for that person, not a loop.

3. A safe place to run, where a bad variant costs nothing. A copy of the data, an agent sandbox, a separate database. If variant ten of three hundred writes to the live ledger, that is not an experiment, that is an incident.

Miss one and the work is not ready to be handed over. The missing piece is almost always the second one.

Where this actually runs today

The setting an ordinary team can reach is model, prompt and retrieval work. Optuna calls a run a study made of trials, where a trial is a single call of the objective function you wrote. You declare what values each setting may take, then give the study a stopping rule: n_trials caps the attempts, timeout stops it after so many seconds, and its pruners automatically stop unpromising trials at the early stages of the training. A DSPy optimiser does the same for prompts: hand it your program, a set of examples and a metric, and it rewrites the instructions and picks few-shot examples to raise that metric, inside a budget you set with max_metric_calls.

The other setting is physical. A self-driving lab hangs the same loop on robots: a model proposes an experiment, a machine runs it, an instrument measures the product, and the result decides what to try next. Berkeley Lab's A-Lab ran 353 experiments over 17 days of continuous operation and realised 36 compounds from a set of 57 targets. That is a research facility with multi-year funding, not something you can buy, and it earns its place here because the loop is the same one.

A worked example: an overnight tuning run

A technical wholesaler runs an internal assistant over its product sheets and supplier datasheets. Answers are fine some days and wrong on others, and nobody can say which setting is to blame.

The score. Two people spend a day writing down 180 real questions and, for each, the document and page holding the answer. That list is the whole basis of the run. They split it: 120 questions the loop may see, 60 locked away that it never touches.

The search space and the budget. Chunk size at 300, 600 or 1000 tokens; overlap at none or 15 percent; 3, 5 or 8 chunks per question; reranker on or off. That is 36 combinations, small enough to try all of them, so the winner is not an artefact of the search method. Times 120 questions it comes to 4,320 scored questions, capped three ways: 5,000 model calls, four hours of wall clock, and a hard spend limit, whichever hits first. The run starts at 22:00 and writes every variant, its settings and its score to a table.

The result. The live configuration, 600-token chunks with no overlap, 5 chunks, no reranker, scores 71 out of 120, so 59 percent. The winner is 300-token chunks with 15 percent overlap, 8 chunks and the reranker on, at 103 out of 120, so 86 percent.

The held-out check. Next morning someone runs only those two configurations against the 60 questions the loop never saw. The old one gets 35 out of 60, so 58 percent; the winner gets 48 out of 60, so 80 percent. The gap has shrunk from 27 points to 22, and that shrinkage is why you held the 60 back: part of the overnight gain was the loop fitting itself to those particular 120 questions.

What it proves is narrow. The right passage was retrieved more often. Whether the assistant then writes a correct answer from it is a different measurement nobody made, eight chunks and a reranker cost more tokens and more waiting than the score had any reason to count, and none of it covers the questions nobody wrote down.

The score is the whole design

Everything the loop does is a search for the cheapest way to raise one number, which makes the choice of number the entire design decision. There are two ways to get it wrong.

A score that is easy to raise without improving anything. Score a summariser on length and you get long summaries. Score retrieval on whether the right document lands anywhere in the top 20 and you get a configuration that returns 20 documents. The loop is not cheating, it is doing precisely what you asked, and closing the gap between the number and the thing you cared about is your job, before the run starts.

The A-Lab shows how sharp this gets. Its score was whether the target came out as the majority phase above 50 percent yield, judged by an automated refinement of the X-ray diffraction pattern. In early 2024 Robert Palgrave at University College London and Leslie Schoop at Princeton reworked that data and concluded the refinement had misidentified phases, and that two thirds of the predicted structures were ordered versions of compounds already known to be disordered. Gerbrand Ceder, who led the A-Lab, replied that a human can perform a higher-quality refinement, and that the point was to show what an autonomous laboratory can achieve rather than what a good human can. Both can be true at once, and the lesson holds either way: the loop was exactly as good as the thing scoring it, and nothing inside it could notice that the scorer was the weak part.

Overfitting to the set you tuned on. The scikit-learn documentation puts the mechanism plainly: because settings can be tweaked until the estimator performs optimally, knowledge about the test set leaks into the model and the evaluation metrics no longer report on generalisation performance. An automated loop does this faster and harder than a person, because it tries hundreds of combinations instead of five. DSPy's GEPA warns about it in the tooling itself: run it with the training set doubling as the validation set and it tells you that you are risking overfitting.

So hold examples back before the run starts, keep them out of every cycle, and use them once, on the two or three configurations that survived. If the winner's lead collapses there, the loop found something about your tuning set rather than about your business. It is the reasoning behind a train-test split, applied to configurations instead of model weights.

Autonomous experimentation versus A/B testing

Both are experiments with variants and a winner. The difference that matters is who or what provides the outcome the run is scored against.

In an A/B test the outcome comes from real customers. You ship variant B to a slice of live traffic and wait for those people to buy, click, cancel or do nothing. Nobody can compute that in advance, which is why an A/B test costs real money when B is worse, needs enough visitors before the difference means anything, and runs for weeks rather than minutes. In autonomous experimentation the outcome comes from data you already hold, so nothing reaches a customer, a wrong variant costs a few cents of compute, and the loop gets through 36 variants overnight. Use the loop to go from 36 configurations to the two worth a real trial, then let customers settle those two.

What to watch out for with autonomous experimentation

Give it a hard budget it cannot argue with. A cap on attempts, on wall clock and on spend. A loop with no ceiling will spend a weekend and a four-figure bill to move a score by half a point.

Write the search space out by hand. The loop tries what you listed and nothing else. An open-ended instruction to go and improve the pipeline is not a search space, it is an agent with production access.

No writes to anything real. The loop reads a copy and writes to its own scratch space. Nothing goes live unless a person applies it deliberately, with the held-out numbers in front of them.

Record every variant, not just the winner. Settings, score, seed, data version, timestamp. That is what experiment tracking exists for, and without it the winning configuration is a number somebody read off a screen at 07:30 and cannot reproduce in November.

Then there is the limit no rule removes. The loop optimises the number you handed it, finds the cheapest route to raising it, and cannot notice that the number was the wrong one. That check stays with a person, and it happens before the run.

Two situations make an overnight loop the wrong tool outright. The first is anything whose outcome only becomes visible weeks later, such as a price change you see back in next quarter's renewals: you cannot compute that score tonight, so there is nothing to loop on. The second is anything where a variant reaching a real customer is the experiment. That is an A/B test, and it needs a named person accountable for what customers see plus a way to roll back, not a scheduler that starts at 22:00.