AI red teaming

What is AI red teaming?

AI red teaming is testing your own AI system by attacking it. You act like a user who is trying to make the system do something it should not, and you record what happens.

The term is borrowed from security, where a red team works through the steps of a real attack to find weaknesses in a system. What AI red teaming looks for is different. A classic red team wants a way in. An AI red team wants an answer the system should have refused, a rule it quietly ignored, or an action it took that nobody authorised.

The distinction matters because your existing security testing will not catch these. There is no exploit, no privilege escalation and no malformed packet. There is a conversation that ends somewhere it should not have.

How an automated run works

Doing this by hand is slow and depends entirely on how creative your tester is on that particular day. That is why the tooling has moved towards automated runs, and understanding the shape of one is useful even if you never operate it yourself.

You start with a set of attack objectives, one per risk category you care about: things you want to check the system refuses to do.

Then you apply attack strategies to those objectives. This is where it gets interesting, because asking directly usually fails. A model's safety training catches the plain version of the question. So the strategies transform the request instead. Microsoft's tooling includes strategies that encode the text in Base64 or Morse, write it backwards, replace characters with lookalike Unicode, put spaces between letters, or convert it to leetspeak. The model still parses it, and the safety layer sometimes does not recognise it.

The more interesting strategies are conversational rather than textual. A multi-turn attack builds context over several messages instead of asking once. A crescendo attack escalates gradually, each message a small step beyond the last, so no single message looks like something to refuse.

The result is scored, and the headline number is the attack success rate: how many of the attempts succeeded out of the total. That is what you track over time and across versions.

What changes when the target is an agent

Testing a model means checking what it says. Testing an agent means checking what it does, and that opens three questions that do not exist for a plain chatbot.

Prohibited actions. Will the agent perform something it should never do, or something that needs authorisation, or something irreversible. The useful framing here is a three-way split: actions that are never allowed, actions allowed only with a human confirming, and actions that are permanent and need both disclosure and confirmation before they run.

Sensitive data leakage. Can the agent be talked into revealing financial, medical or personal data out of the sources and tools it can reach.

Task adherence. Does the agent actually do the job it was given, following the rules and constraints it was given, without wandering off or skipping a required step.

There is a fourth attack type that only exists once an agent reads external content. Indirect prompt injection hides instructions inside an email, a document or a web page that the agent retrieves through a tool. The user asks something innocent, the agent fetches a document, and the document tells it to do something else. Testing for this means putting attacks into the tool output rather than into the user's message.

What automation does not find

Automated runs are how you cover known risks at scale. They are not the whole exercise, and the limitations are worth stating plainly.

The data is synthetic. A run uses generated data and mock tools, which does not look like your real data distribution and does not behave like your real systems.

The scoring is itself a model, which means it is not deterministic and it produces false positives. A reported success rate is a starting point for a human to review, not a verdict.

And automation only tests the attacks somebody thought to encode. The genuinely novel failure, the one specific to your business process and your data, is still found by a person who understands both. Microsoft is direct about this in its own guidance: the strongest approach combines automated tooling that surfaces risks at scale with expert human analysis.

The practical read: automate the regression testing, and keep human red teaming for the parts of your system where a failure would actually hurt.

What to watch out for with AI red teaming

Do not run it against production. Use a separate environment configured like production but with no real data and no real consequences. An attack that succeeds should not be able to send a real email or move real money.

The results are sensitive material. A red teaming report is a list of things that worked. Treat it with the same care as a penetration test report, and think about who inside your organisation needs to read the harmful output verbatim.

One run is a snapshot. Change the model, change the system prompt, add a tool, and the results no longer apply. Attach the run to your release process rather than doing it once before launch.

A low success rate is not an all-clear. It says the encoded attacks in your library mostly failed. It says nothing about the attack nobody in the room thought of.

Red teaming does not replace guardrails. Finding a weakness is not fixing it. Content filters, permission limits and a human at the point of action are what actually reduce the risk in production.