Agent instruction file (AGENTS.md, llms.txt)

What is an agent instruction file?

An agent instruction file is a plain text file, usually Markdown, that you put where an AI agent will find it, so the agent knows how to behave in that place. In a code repository it says how to build and test the project, which conventions apply and what the agent must leave alone. On a website it says what the site is, which pages matter and where the machine-readable endpoints are. The agent reads the file before it starts working, and a person can read it too.

Two families exist, and they solve two different problems.

Repository instruction files are for coding agents. AGENTS.md is the open standard: OpenAI published it in August 2025, and since December 2025 the Agentic AI Foundation under the Linux Foundation stewards it, alongside MCP. The agents.md site counts more than 60,000 open source projects using one. Claude Code reads CLAUDE.md, Cursor reads its .cursor/rules folder, GitHub Copilot reads .github/copilot-instructions.md, and all three also read or import AGENTS.md, so one file can serve the whole team whatever tool each person uses.

Site-level files are for agents that visit your website. The llms.txt proposal by Jeremy Howard, published in September 2024, describes a short Markdown file at the root of a domain with a title, a one-paragraph summary and lists of links to the pages worth reading. In May 2026 Shopify started serving an /agents.md file on every store by default, with /llms.txt pointing to the same content, the largest rollout of the idea so far. The older cousin is robots.txt, which has told crawlers since the 1990s which parts of a site they may fetch.

The shortest way to picture it: a README written for the agent instead of for the new colleague. It sits next to the work, it is versioned with the work, and the person who owns the work maintains it.

How the two families work

In a repository

Every coding agent walks the directory tree looking for its file. Codex starts at the project root and walks down to the folder you are working in, concatenating every AGENTS.md it finds, so a file in services/payments/ adds to the one at the root and wins where they disagree. Claude Code does the same with CLAUDE.md and loads files in subdirectories only when it touches those folders. The practical rule is the one on the agents.md site: the nearest file takes precedence.

On a website

An llms.txt file has a fixed shape. One H1 with the site name, a blockquote with a summary, then H2 sections that list links with a short note each. A section called Optional holds links an agent can skip when it is short on room. Anthropic publishes one for the Claude Code documentation.

Shopify's default /agents.md goes a step further. It lists the store's MCP endpoint and UCP discovery URL, read-only URLs for products, collections and search, and the store's published policies, so a shopping agent can find what it needs without scraping the theme. Since 28 May 2026 a merchant can replace it with a Liquid template of their own.

The difference with robots.txt is what each file is for. Robots.txt, standardised as RFC 9309, says which parts of a site an automated client may fetch, and even that is a request rather than an access control. Llms.txt says nothing about permission; it says where the useful material is.

What belongs in one and what does not

A good repository file is short and specific. Anthropic's guidance for CLAUDE.md is to stay under 200 lines, because a longer file costs more context and the agent follows it less reliably. Codex stops reading AGENTS.md content at 32 KiB by default.

Put in:

• Commands. How to install, build, run and test, written so the agent can run them without asking. "Run composer test before every commit" beats "test your changes".

• Conventions that differ from the defaults. Naming, folder layout, the ORM you use, the migration workflow. The agent already knows the language; it does not know your house rules.

• Off-limits areas. The folder with generated code, the production configuration, the invoice module that has no tests yet and should not be touched without a human.

• Lessons. The mistakes the agent made twice. More on that below.

Leave out:

• Secrets. The file is committed to the repository, read by every teammate and every tool, and sent to a model provider with each session. On a website it is public by definition, and Shopify's own docs warn against putting contact details in /agents.md.

• Anything the agent can work out from the code. Claude Code's /doctor check proposes trims for exactly this: directory listings, dependency lists and architecture overviews go, the rationale and the conventions stay.

• Long procedures. A twelve-step release checklist belongs in a separate file or an agent skill the agent loads when it needs it, not in the file every session starts with.

Filled in, a file for a small Symfony webshop integration looks like this:

# Webshop sync

Symfony 7 app that pushes webshop orders into the ERP nightly.

## Commands
- Install: composer install
- Tests: composer test (must pass before any commit)
- Never run bin/console app:sync:run against .env.prod

## Rules
- Money is stored in cents as integers, never floats
- New ERP fields go through src/Erp/Mapper.php, nowhere else
- Do not edit anything under var/ or migrations/ by hand

Fourteen lines, and the agent knows how to run the tests, which mistake would be expensive, and which folders are not its business.

Instruction file versus system prompt

The two are easy to confuse because both are text the model reads before it does anything. The difference is who writes each one and where it lives.

The system prompt is written by whoever builds the agent product. Anthropic writes Claude Code's, OpenAI writes Codex's, and if you build your own agent on an API you write it in your code. It lives inside the application and the person using the agent never sees it.

The instruction file is written by whoever owns the place the agent works in: the team that owns the repository, the merchant who owns the store. It lives next to the work, in version control or at a URL, and anyone can open it. Claude Code's documentation is explicit about how it reaches the model: the content is delivered as a user message after the system prompt, not as part of the system prompt itself. So the model reads it and tries to follow it, but the product's own instructions still come first, and a direct instruction you type in the chat overrides the file.

Two consequences follow. The file is context, so it spends tokens from the context window in every session, which is why every vendor tells you to keep it short. And it is guidance, not enforcement. If something must never happen, such as a push to the main branch, Anthropic's advice is to use a permission rule or a hook that the client enforces regardless of what the model decides. The file tells the agent what you would like; the settings decide what it can do.

Writing lessons back into the file

Coding agents start every session with an empty memory. Whatever you explained yesterday is gone today, unless it is in the file. That makes the instruction file the place where a team's experience with the agent accumulates.

Anthropic's guidance says when to add a line: when the agent makes the same mistake a second time, when a code review catches something it should have known about the codebase, or when you type the same correction you typed last session. Every, the publication behind the compound engineering plugin, made a loop out of it in December 2025: plan, work, assess, then compound, where the engineer feeds what was learned back into prompts that live in the codebase, so the next feature is easier to build than the last one.

In practice the habit is small. The agent puts a float in a price field; you correct it; you add "Money is stored in cents as integers" to the file. Next week a colleague's session starts with that rule already loaded.

The file needs pruning as much as adding. Anthropic warns that two contradicting rules make the agent pick one at random, and a rule about a module that no longer exists is pure cost. Read the file once a quarter the way you would read an onboarding document, and cut what is no longer true.

What this means for an SME

You do not need to write code to get value from this. Two actions cover most companies.

Ask your software supplier for the repository's AGENTS.md. If they work with coding agents, the file exists or should. Read it. It is the shortest honest description of your project you will get: what it is, how it is tested, which parts are fragile. If the answer is "we don't have one", ask how the agent knows which tests to run. If the file says "no tests", you have learned something about the codebase you paid for. It doubles as a handover document: a new developer, human or agent, starts from the same page.

Put an llms.txt on your website. It is twenty lines of Markdown: your company name, one paragraph on what you do, links to the pages that answer real questions (products, prices, opening hours, support, the pages that explain what you do not do). Whoever manages your site can add it in an afternoon. If you run a Shopify store you already have one; check what it says and remove anything you would not put on a business card.

What to watch out for with agent instruction files

Nobody has proven that llms.txt is read. Google's Search team says Google Search does not use it. In January 2026 John Mueller answered on Bluesky, when asked whether Google publishing one for its own docs was an endorsement, with a flat "no", and in June 2026 he called the file "purely speculative for now", noting that none of the AI systems use it. Google's documentation on AI features says you do not need new machine-readable files to appear in AI Overviews or AI Mode. Anthropic and others publishing an llms.txt for their own documentation says nothing about whether their agents fetch yours. Treat the file as cheap and harmless, not as a ranking lever, and be wary of anyone selling it as one.

The file is an injection surface. A repository file is read by the agent as instructions, so a malicious or careless pull request that edits it changes what every future session does. Review changes to AGENTS.md the way you review changes to a deployment script. The reverse holds for site files: an agent that fetches a stranger's llms.txt is reading text the stranger wrote, and a well-built agent treats it as untrusted content, which is one line of defence against prompt injection.

Stale instructions cost more than none. A rule that references a folder that moved, or a command that no longer exists, sends the agent down a wrong path with full confidence. The file needs an owner.

Too many files, too many tools. A monorepo with a dozen nested AGENTS.md files plus a CLAUDE.md plus Cursor rules produces a context the agent cannot follow. Pick one file as the source of truth and have the others import it; Claude Code's documentation shows a one-line @AGENTS.md import for exactly this.