Query rewriting (RAG)

What is query rewriting?

Query rewriting is the step in a RAG pipeline between the question a person typed and the search that goes to your index. A model reads the question first and turns it into something the search engine can match: it fixes typos, spells out abbreviations, pulls in context from earlier in the conversation, splits a double question in two, or turns "invoices from March" into a date filter. Only then does retrieval run.

It exists because of a gap every search system has. People ask in their own words, mid-conversation, with half the context in their head. Documents are written in the vendor's words by someone who did not know the question. That was the point of the rewrite-retrieve-read work by researchers at Microsoft Research Asia and Shanghai Jiao Tong University in 2023: adapting the query is cheaper than adapting the retriever or the model.

Think of a librarian. You ask "do you have that book about the Antwerp port strikes" and she searches the catalogue for "dockworkers Antwerp 1970s". The rewrite is her translation from how you ask to how the shelves are labelled.

The rewrites that come up most often

Expanding and correcting. The simplest rewrite adds synonyms and full forms and fixes spelling, so "dso" becomes "days sales outstanding, DSO, debtor days". Azure AI Search does this inside its semantic ranker: your query goes to a generative model, up to ten variants come back, and the search runs on the original and the variants together.

Resolving the conversation. In a helpdesk assistant the second question is rarely complete. A customer asks "does the X200 have automatic descaling", gets an answer, then types "and the X300, does it do it too". Searched on its own that finds nothing. A rewrite that reads the previous turn produces "does the X300 have automatic descaling", which your manuals can answer.

Splitting a compound question. "What is the warranty on the X300 and can I extend it" is two searches. A rewrite that makes two sub-queries, retrieves for both and merges the passages answers both halves. When the model also plans the sub-queries and loops until it is satisfied, you are in agentic RAG, which has its own entry.

Several phrasings at once (multi-query). The model writes a few versions of the question from different angles, each is searched, and the results are merged without duplicates. LangChain's MultiQueryRetriever does this, with a default prompt asking for three versions. You pay several searches instead of one, and the answer depends less on how the question happened to be worded.

Searching with a hypothetical answer (HyDE). A question and its answer look different in embedding space. HyDE, published in December 2022 by researchers at Carnegie Mellon and the University of Waterloo, asks the model to write a short passage that could be the answer and searches with that passage instead. The passage contains invented details, as the paper admits, but it looks like the text you want to find and the embedding step smooths those specifics away.

Turning a question into a filter. Part of what people type is a search term and part is a filter in disguise. "What did we agree with Vandenbroucke about delivery" is a customer filter plus a topic query ("delivery terms, lead time"). Passing that structured part to the index as a metadata filter beats hoping the embedding of "March" lands near the right month. LlamaIndex calls it auto-retrieval: from one question the model returns a query string and a list of filters.

What it costs

Every rewrite is a model call before the search starts, so each one adds latency and tokens to every question. Multi-query multiplies the searches, and HyDE makes the model write a paragraph instead of a line. Behind a chat window where someone is waiting that adds up, so a length check that skips the rewrite for short, clear questions pays for itself. Managed features price the same way: Azure's query rewrite runs only with the paid semantic ranker, in a limited set of regions, and is still in preview. When the call fails it falls back to the original query and says so. Build the same fallback if you write the step yourself.

How do you know it helps?

Query rewriting is one of the easiest RAG changes to add, and one of the easiest to add for no gain. The honest test is retrieval recall. Take fifty to a hundred real questions from your logs, including the messy ones, and note which passage should come back for each. Run retrieval twice, on the raw question and through the rewrite, and count how often the right passage lands in the top five.

Keep only the rewrites that move that number. In our experience conversation resolution and filter extraction move it a lot on helpdesk and customer data, synonym expansion moves it a little, and HyDE helps in one collection and hurts in the next.

Azure's debug: queryRewrites option returns the variants next to the results, so you can read what was actually searched. A home-made rewrite step should log the same thing. Without it you cannot tell a bad rewrite from missing content, and those need different fixes.

Query rewriting versus reranking

Both are extra model calls bolted onto a plain search, and the difference is where they act. Query rewriting works before retrieval: it changes what you search for, so it can bring back passages the raw question would never have reached. Reranking works after retrieval: it reorders what came back, so it can only promote a passage that was already found. If the right document is not in the top fifty, no reranker saves it and a rewrite might. If it sits at position twelve, a reranker fixes that and a rewrite is beside the point. Most pipelines end up with both, and the recall test above tells you which your data needs first.

What to watch out for with query rewriting

Rewrites drift. The model can rewrite a question into something the person did not ask. LlamaIndex's own example pulls the filter value "mafia" out of a question while the index stored "Mafia", so nothing matched. Log the original question next to the rewritten query, and show the user which one was searched when an answer looks off.

Exact identifiers get lost. Microsoft warns that rewritten queries may not contain all the exact terms of the original, which matters for article numbers and product codes. Keep the original in the search next to the rewrites, or route anything that looks like a code straight to keyword search.

The conversation can be the wrong context. Resolving "it" from the previous turn assumes the person is still on the same subject. When they switch topics, the old subject comes along. A window of one or two turns is usually enough.