RL environments

What is an RL environment?

An RL environment is a runnable copy of a task that an AI agent can practise on and be scored on. It has a starting state, a set of tools the agent is allowed to use, and a program that decides afterwards whether the attempt worked. Run it a thousand times and you have a thousand graded attempts with nobody watching any of them.

Think of the simulator a haulage firm puts a new driver in. The truck is not real and the roundabout is not real, but the manoeuvre is close enough that a driver who gets good in the simulator is better on the road, and the machine can say whether the trailer hit the kerb.

The RL is reinforcement learning. The model tries something, gets a number back, and shifts towards whatever earned the number. The environment is the half that produces the number. Post-training covers where this sits in the making of a model, so this entry stays on the thing the model is trained against.

The four parts of an environment

The open frameworks make the shape explicit. Gymnasium, the maintained fork of OpenAI's Gym library that the Farama Foundation looks after, gives every environment two methods. reset() puts it back to a starting state and hands the agent its first observation, and it takes a seed so you can get that same starting state again. step() takes one action and returns the next observation, a reward, and whether the episode is over. OpenEnv, the specification Meta's PyTorch team and Hugging Face publish together, keeps those calls and adds what agent work needs: the environment is packaged as a Docker container and served over HTTP, so a training run can start hundreds of identical copies at once.

The starting state. The data, files and systems the task begins from. For office work that is a database snapshot, a mailbox with one message in it, a folder of scanned documents.

The tools. The actions the agent is allowed to take, and nothing beyond them. This list is a design decision rather than a technicality: an agent that can reach the database directly will never learn to use the booking screen.

The grader. The program that reads the end state and returns a score. Epoch AI, writing in January 2026 after interviewing people who build these for a living, put the whole thing in one sentence: each task is a prompt telling the model what to achieve plus a grader that determines whether the objective was met.

The reset. Everything the attempt touched goes back exactly as it was, thousands of times a day, with no leftovers. That sounds like plumbing and it is the reason environments get packaged as containers.

Why environments became the scarce part

Models improved fastest on work a program could grade. Maths has a right answer, code either runs or it does not, and that is where the training signal was cheap and unlimited. So the scarce resource moved. It used to be text. Now it is well-built tasks that a machine can mark.

Several people have said this out loud, with dates on it. Mechanize, which builds environments and evals for coding agents, argued in June 2025 that getting further would need training environments at a scale and diversity that dwarf anything currently available. Prime Intellect opened a public hub for open environments in August 2025 with a blunter version: environments are the key bottleneck to the next wave of AI progress, and the big labs are locking them down. In September 2025 TechCrunch went round the whole scene and quoted Jennifer Li of Andreessen Horowitz saying that every big lab is building environments in house and that creating the datasets is very complex. The same piece cited reporting by The Information that Anthropic had discussed spending more than a billion dollars a year on them.

Nobody sells this to a company your size. The customers are the labs, and Epoch's interviews put the price of a single task somewhere between two hundred and two thousand dollars, with exclusive deals four to five times that. What you can look at for free is the open end: Gymnasium, OpenEnv and the environments people publish on Prime Intellect's hub are all open source.

What separates a useful environment from a useless one

A score that cannot be gamed. A model optimising against a program learns to make that program say yes, and the shortest route there is often not the honest one. Epoch's summary of what builders worry about first: high reward must mean the task was actually solved, not hacked, and it takes many iterations to get there. Ross Taylor, a former research lead at Meta AI, told TechCrunch that people are underestimating how difficult it is to scale environments, for exactly this reason.

A task close enough to the real one. Skill only transfers if the simulation behaves like the thing it copies. One of Epoch's interviewees was rude about the common failure: there are good reasons to use clones of websites, but what everyone does is vibe code a buggy website, which is not useful. An agent that gets good at working around your clone's bugs has learned your clone.

A reset that actually resets. If attempt 400 starts with residue from attempt 399, part of your training signal is measuring the residue.

Enough variation, at the right difficulty. One instance teaches the instance, and the worked example below shows how that goes wrong. Difficulty cuts the other way: a task the model never solves teaches nothing, so builders aim for a floor of a few percent, at least one success in sixty-four or a hundred and twenty-eight attempts.

A small environment built from one business task

Take booking an incoming supplier invoice against a purchase order at a wholesaler. Here is what turning that into an environment involves.

Starting state. A snapshot of the ERP database with four hundred open purchase orders and sixty suppliers, plus one PDF invoice sitting in an inbox folder. The same snapshot is restored before every attempt.

Tools. Four functions and nothing else: search_purchase_order, read_document, create_booking and flag_for_review. No shell, no direct SQL.

Score. After the attempt, compare the database against the answer recorded when the task was written. One booking row, linked to purchase order 4471, on the right general ledger account, with the right VAT code and the amount from the PDF. Anything else scores zero. This is how tau-bench grades agents, the open customer service benchmark Sierra Research published in 2024: it compares the database state at the end of the conversation with an annotated goal state.

Variation. Eight hundred versions of the same task. Different suppliers and amounts, some with no matching order, some with a duplicate invoice number, some just outside tolerance. In one batch the invoice is 4182.30 euro against an order of 4100.00 euro, which is 2.01 percent over, and the policy allows two. The right answer there is to flag it, not to book it.

Then comes the interesting part, the ways an agent scores well without doing the job.

1. It finds the answer instead of the invoice. The recorded goal state is a file, and files live somewhere. If it is reachable inside the container, or if the order number turns up in a log line, a capable agent will read it.

2. It flags everything. Give partial credit for flagging and the winning strategy is to flag every invoice and book none. The score goes up and no work happens.

3. It learns the generator, not the task. If whoever wrote the eight hundred versions always used the most recent open order for that supplier, the model learns to pick the most recent open order. Full marks in the environment, and it fails on the first invoice that arrives late.

4. It writes the row. One tool with a wider write path than intended and the agent sets the target row directly, skipping the matching altogether.

None of these are exotic. They are what you get by default, and hunting them down is most of the work.

An RL environment versus an evaluation set

They are built the same way. Prime Intellect's own documentation says it flatly: RL environments and agent evals are basically the same thing, a dataset plus a harness plus scoring rules. Mechanize describes its graders the same way, as producing scores that act as signals during reinforcement learning and during evaluation. What differs is what happens to the score afterwards.

An evaluation set is read-only for the model. The agent runs it, the score is recorded, and nothing about the model changes. Its entire worth comes from the model not having practised on it. Use it twice a year on every version you weigh up and it keeps telling you something.

An RL environment is something the model learns from. It runs thousands of times and the weights move on the result. Its worth comes from the repetition, and from the model getting better at it.

The consequence is the part people skip. Once a model has trained on an environment, its score there stops being a measurement. A high number says the model learned this environment, which may or may not mean it learned the task. That is why labs hold sets back, and why a benchmark score in a release note tells you less the more famous the benchmark is.

For you it comes down to one small discipline. Keep twenty of your own real cases with the answers you expect, never publish them, and never hand them to a vendor as fine-tuning material. The moment they leave your building they stop being a test.

What to watch out for with RL environments

They explain why a model is good at generic office work and poor at yours. Improvements land where the training signal was. Somebody built environments for coding, for browsing, for customer service in an airline domain. Nobody built one for your intercompany allocation rules, so nothing in the next release is aimed at them.

A demo on a simulated version of your process is evidence, not proof. The simulation contains what somebody wrote down. The reason your invoice clerk is good is mostly the part nobody wrote down: that this supplier always splits the delivery, that the March file is wrong every year, that when Kris says approved he means approved once the credit note lands. An agent that scores well against the written half has demonstrated the written half.

A score can be true and useless at the same time. Ask two questions about any number you are shown: what exactly did the grader check, and over how many variations. A number from one scripted run tells you the run worked once.

This is not a market you buy into. Mechanize, Mercor, Surge and Scale AI sell to labs, not to you, and the open frameworks are there to be read rather than installed. Reading a published environment is still worth an afternoon, because seeing how somebody wrote a grader for work like yours changes how you write your own acceptance criteria.