Prompt versioning

What is prompt versioning?

Prompt versioning is treating every prompt in your AI application the way you treat code: each change gets a number, a date, a reason and a name, the old version stays available, and you can put it back when the new one turns out worse. It covers system prompts, prompt templates, few-shot examples, tool descriptions and agent instruction files. If the model reads it before it answers, it is behaviour, and behaviour needs a history.

A prompt is not documentation, it is the program. Change "summarise the complaint" into "summarise the complaint briefly" and a share of your outputs change with it, in ways you only notice once you go and look. Nobody remembers on Friday what was edited on Tuesday, and the same prompt often runs in three places at once: the chatbot, the nightly batch job, and the Power Automate flow a colleague pasted it into.

Google's SRE book made the same argument about configuration years ago. Configuration changes are a potential source of instability, so configuration belongs in the source repository under a strict review requirement. A prompt is configuration that happens to be written in English or Dutch.

What you put under version control

A version that says "v5" without saying what it was tested against tells you nothing when something goes wrong. Store with the text:

• The variables: which placeholders the template expects and in what format. Rename {{ticket}} to {{ticket_text}} and every caller breaks.

• The model ID it ran against, in its pinned form. Anthropic's documentation says every Claude model ID is a pinned snapshot, and OpenAI's guidance is to pin production applications to specific model snapshots so behaviour stays consistent.

• The settings: parameters, output schema, the tools that were switched on. Anthropic deprecated temperature from Claude Opus 4.7 onwards, so a prompt that still sends a non-default temperature to one of those models gets an error back.

• The eval score, the test set behind it, and who approved the change. That turns "I think it is better" into "90 percent on the same 120 cases where v4 scored 76".

Where the versions live

In the repository, next to the code. The prompt is a plain file, prompts/invoice-extract/v5.md, and Git does the rest: history, diff, review, rollback. OpenAI moved this way in 2026. It began winding its reusable prompt objects down on 3 June 2026, the v1/prompts endpoint is scheduled to shut down on 30 November 2026, and the advice is to keep prompt builders in a small module near the feature they support.

In a prompt registry inside an observability or gateway tool. Langfuse, PromptLayer and Braintrust keep prompts as named objects with a version number and labels such as production and staging. The application fetches by name and label at runtime, so a rollback is moving the production label back a version, with no deployment, and traces in the tool can record which version produced an answer. The catch is that the prompt now changes without a code change. Langfuse and PromptLayer both let you protect a label, so only an admin or an approval flow can move it.

If only engineers edit prompts, keep them in Git. If a subject-matter expert edits them weekly, a registry earns its place, as long as that label is protected and every change runs the eval set first.

From edit to production

1. Change on a branch or as a new draft version, never in the live one.

2. Run the eval set against the same pinned model the current version uses. Anthropic's advice is to have your success criteria and a way to test against them before you start improving the prompt, and to prefer many automatically graded cases over a handful graded by hand.

3. Compare per field, not on the overall score. A prompt that gains ten points on one field and loses two on another is a decision, not a win.

4. Review the diff with someone who owns the outcome. Prompt diffs are short and readable, which makes this the cheapest review in the stack.

5. Deploy and log one line: what changed, the scores, the model, who approved it. Keep the previous version reachable.

A version bump, worked through

An invoice extraction prompt pulls supplier, date, currency and total out of scanned PDFs, on claude-haiku-4-5-20251001, against 120 real invoices. Version 4 says "Return the total amount." Version 5 says "Return the total amount including VAT, as a number with two decimals, in the invoice currency."

Total goes from 91 correct out of 120 to 108, so 76 to 90 percent. Currency stays at 118. Date slips from 112 to 110, because two invoices now return the due date. That regression goes in the changelog and gets fixed in version 6:

2026-09-02  invoice-extract v5
  Change: total includes VAT, 2 decimals, invoice currency.
  Eval (120 invoices): total 76% -> 90%, currency 98%, date 93% -> 92%.
  Model: claude-haiku-4-5-20251001. Approved: An (finance).

Six months later, when a customer asks why a March invoice was booked with the wrong date, that line is the whole investigation.

Prompt versioning versus prompt engineering

Both are about changing prompts, so they blur. The difference is what each one produces. Prompt engineering produces a better prompt right now: you rephrase, add an example, restructure with XML tags, and the output improves. Prompt versioning produces a history you can roll back, so you can say what changed, when, tested against what, and how to undo it. A team can be excellent at the first and have no answer to the second, and that is the team that hears from a customer that last month's clever rewrite broke the French invoices.

Model versioning is the third piece. Anthropic commits to at least 60 days of notice before retiring a publicly released model, and a prompt that scored 90 percent on the old model has no score on the new one until you run the evals again. Treat a model upgrade as a version bump with an empty diff.

What to watch out for with prompt versioning

The edit in the vendor portal that nobody logged. Someone tweaks the system prompt in a chatbot builder at half past five to settle one complaint, and next morning every answer sounds different. If a tool lets people edit a live prompt, either lock that screen or make it the only place prompts get edited, with its own history.

The eval set that never runs. A folder of test cases is not a safety net. Wire the eval into the pipeline that deploys the prompt, so nothing reaches production without a score next to it.

Ten copies drifting apart. The same triage prompt gets pasted into the chatbot, an Azure Function, two Power Automate flows and a colleague's notebook, and each copy gets fixed separately. Keep one source and have every consumer fetch it by name and version. Watch aliases too: on some platforms an alias is a pointer to a dated model ID that moves when a newer model lands.

The SME version, no tool required

A company with two or three AI features needs nothing it does not already have: a prompts/ folder in the application repository with one subfolder per prompt, a naming rule such as <name>/v<number>.md with the model ID in a header, one evals.jsonl per prompt holding thirty to a hundred real cases with their expected answers, and one changelog line per change in the shape above. Add an eval case whenever something goes wrong in production, and let no prompt reach production without that line and a second person's initials. At twenty prompts and five editors, move the same folder into a registry. The history comes with you.