Graph RAG

What is Graph RAG?

Graph RAG is retrieval-augmented generation where the retrieval step walks a graph of entities and the relations between them, instead of only pulling the text chunks that look most like the question. The model still writes the answer. What changes is what it gets handed: not five loose passages, but a connected set of facts, and sometimes a summary of a whole cluster of documents.

Plain vector RAG works well when the answer sits in one passage. It struggles when the answer only exists once you connect things that are spread across many documents, or when the question is about the corpus as a whole. Nobody wrote a paragraph that says which suppliers deliver to the plants that had quality incidents last quarter. That fact is a path through three tables, or through thirty documents, and a similarity search on chunks has no way to follow a path.

A graph does. Supplier delivers to plant, plant had incident, incident dated last quarter. Graph RAG builds or uses that structure and lets the model reason over it.

Two families

The name covers two approaches that share a word and little else. It pays to know which one someone means.

A graph extracted from documents. Microsoft Research published this version in 2024 and released it as the open-source GraphRAG library that July. During indexing an LLM reads every chunk and pulls out entities, the relations between them and short descriptions. Those become nodes and edges. A community detection algorithm (Leiden) then groups tightly connected entities into clusters, in several levels, and the LLM writes a report for every cluster. At query time a local search starts from the entities in your question and fans out to their neighbours. A global search reads the cluster reports instead of the documents and combines them into one answer. That global mode is the reason this family exists: it is how you answer "what are the main themes in this year's customer complaints" over a few thousand tickets. Microsoft's own example was a news corpus about the war in Ukraine, where baseline RAG could not answer a question about a named group because no single passage discussed it, while the graph connected the scattered mentions.

A graph you already have. Most companies do not need to extract a graph from prose, because the structure already sits in their systems: customers, orders, products, sites, technicians, parts. Load that into a graph database, or leave it in place and describe the schema, and let the model write a graph query. Neo4j's GraphRAG package ships a Text2Cypher retriever that turns a question into a Cypher query, runs it and hands the rows to the model, and a vector-then-Cypher retriever that finds a starting node by similarity and then traverses from it. LlamaIndex's property graph index does the same with a TextToCypher retriever and with Cypher templates where the model only fills in the blanks. If your graph is RDF, the query language is SPARQL rather than Cypher and the idea is identical.

The second family is closer to text-to-SQL than to the first family. It is cheaper, more predictable and easier to audit, because the graph is your data and not a model's reading of your documents.

The questions plain vector RAG cannot answer

Two shapes of question break chunk retrieval, and both are common in a business.

Multi-hop questions. "Which suppliers deliver to the plants that had quality incidents last quarter?" The answer needs a supplier list per plant, an incident list per plant and a date filter, joined together. No passage contains all three. Vector search returns chunks about suppliers and chunks about incidents, and leaves the joining to a model that cannot see the rows it did not receive.

Global questions. "What are the main themes in this year's complaints?" There is no passage to retrieve, because the answer is a property of the whole set. Vector RAG returns the ten complaints that resemble the word "theme" and summarises those. The Microsoft team measured this on two corpora of roughly one and 1.7 million tokens. Judged on how complete and how varied the answers were, their graph approach was preferred over vector RAG in most head-to-head comparisons, between about 62 and 83 percent depending on the corpus and the criterion.

A question that names one thing and wants one fact ("what is the warranty period on the X200") does not need any of this, and a graph will not make that answer better.

Example: an installer's service history

An installation company has ten years of service data: tickets, the technician who handled each ticket, the site, the part that was replaced, and which team installed the site in which year. All of it lives in a ticketing system and an ERP.

Someone asks: "Which part fails most often at sites installed by team B in 2024?"

With chunk retrieval you get tickets that mention team B and tickets that mention frequent failures. Team B is rarely named on a ticket, because the installation happened years before the ticket was opened. The answer is a count across a path, and the path looks like this in Cypher:

MATCH (:Team {name: 'B'})-[i:INSTALLED]->(s:Site)
WHERE i.year = 2024
MATCH (s)<-[:AT_SITE]-(t:Ticket)-[:REPLACED]->(p:Part)
RETURN p.name, count(t) AS failures
ORDER BY failures DESC
LIMIT 5

The model's job is to write that query from the question and the schema, run it, and put the top row into a sentence. Every number in the answer can be traced back to a ticket. That is the second family, and for a company whose data already sits in tables, it is the one to start with.

The first family enters when the question becomes "what do technicians keep writing in their notes about the X200", because that is prose, and the entities (part, symptom, cause) have to be pulled out of free text first.

Vector RAG versus Graph RAG

The dimension that decides between them is the shape of the question: does the answer live in one passage, or in the connections across many?

Vector RAG answers questions whose answer sits in one or a few passages. Retrieval is a similarity search, indexing is one embedding call per chunk, and the pipeline is cheap, fast and well understood. When the answer is present in the text, this wins on cost and simplicity, and hybrid search plus reranking closes most of its gaps.

Graph RAG answers questions whose answer only exists once you connect entities across documents or tables, and questions about the corpus as a whole. Retrieval is a traversal or a graph query. Indexing either costs an LLM pass over every chunk or reuses structure you already have. The answer comes with an explicit path you can check.

Most production systems that use a graph keep the vector index next to it. Neo4j's package and Microsoft's library both start many queries with a similarity search to find an entry point, and then walk the graph from there.

What it costs

The expensive part is extraction. In the document-graph family an LLM reads every chunk at indexing time, and then reads the graph again to write the cluster reports. Microsoft's repository opens with a warning that indexing "can be an expensive operation" and tells you to start small. In its paper the team indexed a corpus of about one million tokens of podcast transcripts, and that graph indexing run took 281 minutes, close to five hours, on one machine.

The clearest number comes from Microsoft itself. When it announced LazyGraphRAG in November 2024, it stated that LazyGraphRAG's indexing cost is identical to vector RAG and 0.1 percent of full GraphRAG. Read the other way round: indexing with full GraphRAG costs on the order of a thousand times what embedding the same chunks costs. LazyGraphRAG gets there by skipping the LLM at indexing time, building a concept graph from plain noun-phrase extraction, and spending model calls only at query time, on the part of the graph a question touches. Microsoft ships it inside Microsoft Discovery, its research platform on Azure. In the open-source GraphRAG library it has been named as the next milestone since December 2024 and has still not appeared in a release, and Microsoft now describes that repository as largely in maintenance mode.

Query cost also differs by mode. A global search reads every cluster report through a map-reduce, so one question can cost hundreds of model calls. Microsoft's later dynamic community selection cut that by 77 percent on average by dropping reports that have nothing to do with the question. A local search or a Cypher query costs about what a normal RAG question costs.

The second family has no extraction bill. You pay for a graph database or for a schema description, plus one model call to write the query.

What to watch out for with Graph RAG

Extraction errors propagate. If the LLM reads "Peeters BV" as a person, or misses a relation, that error is now a fact in the graph and every answer that walks over it inherits it. Microsoft's own team showed that its default extraction prompts, written for news, miss what a chemist expects from chemistry papers, and built an auto-tuning step that generates domain prompts. Budget time for reading a sample of the extracted graph before you trust it.

Entity resolution decides the result. "Team B", "installation team B" and "TB" have to become one node, or the count in the example above is wrong. Neo4j's graph builder ships exact-match, fuzzy and embedding-based resolvers for a reason. On structured data this is the same master data problem you already have. On extracted text it is harder.

Graphs go stale. A vector index can re-embed a changed document. An extracted graph has to re-run extraction and, in Microsoft's design, recompute the clusters and their reports. Decide how updates flow in before you build.

Generated queries can be wrong. A Cypher query that runs is not a Cypher query that is right. Show the query to the user, or constrain the model to templates, the way LlamaIndex's Cypher template retriever does.

When not to bother. A few hundred FAQ documents, product manuals, an HR policy: single-passage questions on a small corpus. Vector RAG with hybrid search handles that, and a graph adds cost and a new source of errors without a better answer. Start a graph when you can write down three real questions that need a join across documents, or one global question you have to answer every month.