Agentic RAG

What is agentic RAG?

Agentic RAG is retrieval-augmented generation with a planning step in front of it. In classic RAG your question goes straight to the search index, the closest matching chunks come back, and the model writes an answer from them. In agentic RAG a model first looks at your question and decides how to go and find the answer.

The reason this exists is that one search does not cover one question. Take something like "find me a hotel near the beach, with an airport shuttle, and within walking distance of a vegetarian restaurant". That is three separate requirements in one sentence. A single search across your index will match on some of the words and miss the rest. Split it into three searches and you can actually answer it.

The same applies to questions that lean on what was said earlier in the conversation, and to questions with a typo in them. An agent that reads the whole conversation before it searches can rewrite "and what about the other one" into a query that will actually find something.

What the agent adds to ordinary retrieval

Azure AI Search implements this as a named pipeline, and the four steps it runs are a good description of what agentic RAG means anywhere.

1. Query planning
   Your question and the conversation history go to a language model, which writes a set of focused subqueries. This is where a compound question gets taken apart and where context from earlier turns gets folded in.

2. Parallel execution
   All the subqueries go to your knowledge sources at the same time. Each one can be a keyword search, a vector search or a hybrid of the two, so you are not locked into one retrieval style for the whole question.

3. Reranking
   Each subquery gets its results reordered by relevance before anything moves on. That matters because a subquery that finds fifty candidates is not helpful unless the good ones end up on top.

4. Merging
   The system pulls everything together into one set of grounding material. You can also ask for the source references and an activity log alongside it, so you can see which queries were sent to which source.

That last point deserves attention. The activity log is what turns agentic RAG from a black box into something you can debug. When an answer is wrong, you can look at the query plan and see whether the model asked the wrong question or whether your content simply did not contain the answer. Those are two completely different problems with two completely different fixes.

Some setups add a further step where the model criticises its own answer and searches again if it is not good enough. That is where the word agentic really earns itself: the system decides for itself that it is not done.

How it compares to classic RAG

Classic RAG is one question, one search, one answer. It is fast, it is cheap, it is easy to reason about, and it is the right choice for most straightforward lookups. If your users ask "what is our warranty period on product X", you do not need a planning step.

Agentic RAG handles the questions classic RAG quietly fails at: several requirements in one sentence, a follow-up that only makes sense in context, or a question where the answer is spread over several documents that no single search will pull together.

The trade is straightforward. Agentic RAG adds latency compared to a single-query pipeline, and it adds cost, because you are now paying for a model to plan on top of paying for the search itself. Microsoft says so plainly in its own documentation. What you get back is the ability to handle a kind of complexity that one query cannot.

What it costs

Something changes in how you budget, and it catches people out. Classic search bills you per query, with a predictable cost per query. Agentic retrieval bills you per token, and the cost per question varies with how much planning the model does.

In the Azure implementation you get two bills. The search service charges for the tokens used while running subqueries and reranking, and Azure OpenAI charges for the input and output tokens used by the model that plans the queries. Estimating your cost therefore means estimating token volume, not counting queries.

There are three levers for keeping that under control, and they are worth knowing before you build.

• Reasoning effort. The pipeline has effort levels. At the lowest setting the planning step is skipped entirely and your query goes straight to the sources, which turns the whole thing back into classic retrieval. Higher settings buy you more planning and more query expansion.

• Number of knowledge sources. Every extra source widens the fan-out, so consolidating your content reduces token volume directly.

• How your content is organised. If the answer to a common question sits in one curated summary instead of scattered over forty documents, the pipeline finds it with fewer subqueries.

What to watch out for with agentic RAG

It does not fix bad content. A planning step makes your retrieval smarter, not your documents better. If your index holds three contradictory versions of the same policy, agentic RAG will find all three faster.

Latency is noticeable. Planning, several searches and reranking take longer than one lookup. Behind a chat window where the user is waiting, that difference is felt. Consider whether the hard questions justify slowing down the easy ones, or whether you route simple questions down a cheaper path.

Permissions have to survive the fan-out. Several subqueries against several sources means several chances to return something the user should not see. The filtering has to happen at the source and not in the final answer.

Read the activity log before you tune the prompt. When quality disappoints, the reflex is to rewrite instructions. Look at which subqueries were actually generated first. Very often the plan was reasonable and the content was the problem.

Cost scales with question difficulty, not with usage. Two hundred simple questions can cost less than twenty complicated ones. Any budget you build on an average cost per question will be wrong in both directions.