Parametric knowledge

What is parametric knowledge?

Parametric knowledge is what a language model absorbed into its weights during training. It arrives with no source attached. The model does not keep the document a fact came from, only a statistical trace of having seen that pattern often enough, so it can produce the fact but cannot point at where it read it.

Contextual knowledge is the other half: whatever sits in the prompt at the moment you ask. A pasted contract, a retrieved paragraph from your intranet, a row pulled from your data warehouse. That knowledge does have a source, because you handed it over and you know what you handed over.

The 2020 paper that introduced retrieval-augmented generation named the split when it described its architecture: the parametric memory is the pre-trained model itself, and the non-parametric memory is a searchable index the model reads from at answer time. Grounding, citations and tool calling are all ways of shifting weight off the first and onto the second.

How you tell the two apart

The tell is confidence without traceability. Ask an assistant how VAT works on a Belgian invoice and it answers smoothly, because tax writing is all over the training data. Ask the same assistant what your own company charges on a specific service line and it answers just as smoothly, in the same tone, with nothing behind it.

Fluency is not a signal here. Both answers cost the model the same effort and sound equally settled.

Three checks separate them in practice.

• Ask for the passage. Not a summary of the source, the sentence itself. Parametric knowledge cannot produce one, so the model either declines or invents a plausible looking reference.

• Change the document and ask again. If you edit the underlying file and the answer does not move, the answer was never reading that file.

• Ask something only your source can answer. An internal project code, a customer-specific discount, a clause number. Training data has never seen these, so a fluent answer is a fabrication.

When retrieved context contradicts what the model learned

This is the case most teams never test, and it is where the two kinds of knowledge collide. Research on the topic calls it a context-memory conflict: the retrieved document says one thing, the weights say another.

The direction of the published findings is that models lean hard toward the retrieved text. In one benchmark built specifically to measure this tug-of-war, leading models dropped their own correct answer and repeated the wrong document well over half the time. Two things moderated that: the more obviously implausible the retrieved text was, the more the model resisted it, and the less confident the model was in its own answer, the more readily it deferred.

That combination cuts both ways for a business assistant. A model that defers to your documents is doing what you want when the documents are right. When someone uploads last year's price list, the same behaviour propagates the stale price with a citation attached to it.

The opposite failure is quieter. When the retrieved chunk is thin or off-topic, the model can fall back on training and answer anyway, which is why Azure OpenAI On Your Data ships a setting for it. Microsoft's documentation is direct about the default behaviour: with the limit-responses-to-your-data flag off, "the model supplements its responses with its own knowledge in addition to your documents".

Parametric knowledge, fine-tuning and RAG

Fine-tuning writes into the weights

Fine-tuning adjusts the parameters, so whatever it teaches becomes parametric knowledge. That means it inherits the properties: no source, no citation, and no way to update one fact without another training run. Fine-tuning is a good lever for style, format and task behaviour, and a poor lever for facts that change.

RAG leaves the knowledge outside

RAG keeps the facts in a store you control and pulls them in per question. Changing a fact means editing a document, not retraining. That is why company knowledge, prices and procedures belong on the retrieval side of the line and almost never in the weights.

Citations sit on top of the contextual half

A citation is only possible when there is a passage to point at, which means citations only exist for contextual knowledge. Anthropic's citations feature returns the exact source sentences behind each claim, and its documentation notes that because the API extracts that text directly, the pointers are guaranteed to be valid ones into the documents you supplied.

Read that guarantee carefully. It says the passage is real and really was in your document. It does not say the sentence built on that passage is a fair reading of it. "The model knows it" and "the model can cite it" are different claims, and only the second one is checkable.

What to watch out for with parametric knowledge

It goes stale on a fixed date and never says so. Training data stops at a point in time, and the model has no internal sense of how old any given fact is. A rule that changed after that point still comes out in the present tense, which is one of the ways an answer gets reported as a hallucination when it was really just out of date.

Software facts age fastest. Ask about a feature in Power BI, Fabric or any product on a monthly release cycle and you are likely to get a description of how it worked when the model was trained. The answer is not invented, it is just from an older version, which makes it harder to spot than an outright fabrication.

It is still worth having. Parametric knowledge is what lets a model understand your question at all, follow an instruction, or write correct SQL against a schema it has never seen. The goal is not to suppress it. The goal is to keep it doing language and reasoning, and keep company facts on the contextual side where you can point at them.