Data poisoning

What is data poisoning?

Data poisoning is deliberately manipulating the data a model learns from, so that the finished model behaves the way the attacker wants. OWASP describes it as manipulation of pre-training, fine-tuning or embedding data in order to introduce vulnerabilities, backdoors or biases.

What makes it different from most attacks is the timing. There is no moment where somebody breaks in. The attacker contributes something to the material, and then waits. The damage is baked into the model during training, and it is present from the first day the model is used.

That also makes it hard to detect. There is nothing anomalous to catch in your logs. The model is behaving exactly as it learned to behave. It just learned from something that was arranged for it.

Where in the chain it can happen

OWASP lists four stages, and they are worth walking through because your exposure is different at each one.

Pre-training. The material a foundation model is built on, largely scraped from the public web. Anyone who can put text on the internet can contribute to it. This is far outside your control, and it is the reason to care who built the model you use.

Fine-tuning. Data used to adapt a model to a specific task. This one is inside your control, and it is exactly where a poorly reviewed dataset from a supplier can do damage.

Embeddings. The step where your text is converted to vectors. Corrupt this and you affect what your retrieval finds, which affects every answer built on it.

Transfer learning. Reusing a pre-trained model for a new task. You inherit whatever was in the original, including anything that was put there deliberately.

Backdoors and sleeper agents

The version of this that keeps security people awake is the backdoor.

Poisoning can be used to plant behaviour that does not show up in normal use. The model behaves entirely normally until a specific trigger appears in the input, and then it does something else. OWASP calls a model in that state a sleeper agent, and the description fits: it looks fine, it tests fine, and it is waiting.

The reason this is nasty is that all your usual quality checks pass. You can evaluate the model on a thousand test cases and see nothing, because the trigger is not in any of them. Detection is genuinely difficult, which is why the practical defences are all about what went in rather than about catching the behaviour afterwards.

Why this concerns you even if you never train

Most companies reading this will never fine-tune a model, and it is tempting to file this under someone else's problem. Two things bring it back to you.

The first is the model you chose. Using an open-weight model from an unknown source, or a fine-tuned variant somebody published, means trusting a training process you did not see. That is a supply chain decision, and it deserves the same scrutiny as any other dependency you pull in.

The second is closer to home. Your retrieval index is learning material in every way that matters. If your assistant answers from a document library, then whoever can add a document to that library can influence the answers. A supplier who uploads a document with favourable terms stated as fact, an ex-employee who edits a policy page, an incoming email that ends up indexed: none of that is training in the technical sense, and all of it changes what your system tells people.

That framing is the useful one for a normal organisation. Ask who can put text where your AI will read it. That list is usually longer than people expect.

What you do about it

Know where your data came from. Data lineage is the core defence here. If you cannot say where a dataset or a document originated, you cannot assess it. OWASP points at tooling for this, including a bill of materials for machine learning components, which is the same idea as a software bill of materials.

Vet suppliers and their datasets. A dataset delivered by a third party deserves the same review as third-party code, and usually gets much less.

Version everything. Versioned datasets and versioned indexes let you answer the question of what changed between the run that was fine and the run that was not.

Control who can write to your index. This is the practical one. Adding a document to a retrieval index should be a governed action with an owner, not something anyone can do by dropping a file in a folder.

Test adversarially. Red teaming and stress testing are how you probe for behaviour you did not intend, including behaviour that only appears under specific conditions.

Watch the training signals. If you do train or fine-tune, unusual patterns in training loss are one of the few signals that something in the data is off.

What to watch out for with data poisoning

Not every bad answer is an attack. Contradictory documents, outdated policies and duplicated records produce the same symptom as poisoning and are vastly more common. Check the ordinary explanation first.

Evaluation sets do not catch backdoors. Your test questions measure what you thought to ask. A trigger you never imagined will not appear in them.

Fine-tuning on your own data changes your risk position. The moment you train, the quality and the provenance of your own data become a security question and not just a quality question.

A model published as a helpful variant is still an unknown. Downloading a fine-tuned model from a public hub is convenient, and it is also accepting a training process you cannot inspect.

The retrieval index needs an owner. Someone has to be responsible for what is in it, when it was added and by whom. Without that, you have a shared folder that answers customer questions.