Chunking

What is chunking?

Chunking is the step where a document is split into smaller pieces before those pieces are embedded and indexed for retrieval. Instead of storing a fifty-page policy as one large item, a RAG system stores sections, paragraphs, or token-sized blocks as separate chunks.

Each chunk gets its own embedding. When a user asks a question, the system searches for the chunks that are closest to the question and sends only those pieces to the language model. Good chunking is one of the reasons a RAG chatbot finds the right paragraph instead of vaguely matching an entire manual.

Why documents need to be chunked

There are two practical reasons.

Embedding quality
An embedding turns text into a vector that represents its meaning. If the text covers five unrelated topics, the vector becomes a blurry average. A chunk that covers one clear idea is easier to retrieve accurately.

Context limits
At answer time, the model can only receive a limited amount of context. Sending entire documents is expensive and often worse for quality. Sending a few relevant chunks gives the model less noise and more signal.

Chunking happens before the user asks a question, when the knowledge base is prepared.

Common chunking strategies

Fixed-size chunks
Split the text every set number of tokens or characters. This is easy to automate and predictable, but it can cut through a sentence, list, or table.

Structure-aware chunks
Split on natural boundaries such as headings, paragraphs, Markdown sections, PDF sections, or HTML blocks. This usually preserves meaning better than blind fixed-size splitting.

Overlapping chunks
Repeat a small amount of text from the previous chunk at the start of the next one. Overlap helps when an important sentence sits near a boundary.

Semantic chunking
Split where the topic changes. This can use embeddings or other analysis to group conceptually similar text. It often gives cleaner chunks, but costs more processing upfront.

Most real systems mix these methods: respect headings where possible, keep chunks within a target size, and add a little overlap at boundaries.

Example: an HR policy

Imagine a chatbot on a thirty-page HR policy. The document covers holidays, sick leave, remote work, expenses, and parental leave.

If you embed the whole document as one item, a question about sick notes must match against everything. If you cut blindly every 500 tokens, the rule about sick notes may be split from the paragraph that explains when it applies.

A better approach is to split by section heading, then trim or split long sections further. The chunk about sick leave keeps the rule, exception, and source heading together. When an employee asks about sick notes, the retriever can find that exact passage and the chatbot can answer with the right context.

Choosing a chunk size

There is no universal best size. Smaller chunks give sharper retrieval but may lose context. Larger chunks preserve context but can dilute the match and fill the prompt with irrelevant text.

Common starting points are a few hundred tokens per chunk, with overlap where the content is narrative. Short product descriptions, legal clauses, support articles, and technical manuals all need different treatment.

The right answer is empirical: create a test set of real user questions, try several chunking strategies, and measure whether the retrieved chunks actually contain the answer.

What to store with each chunk

A chunk should not be stored alone. Keep metadata with it:

• Document title so the passage has context.

• Section heading so the model knows where the text came from.

• Source URL or file path so the answer can cite the source.

• Version or date so old policies can be expired or replaced.

• Permissions so users only retrieve content they are allowed to see.

What to watch out for with chunking

Changing strategy means reindexing
If you change chunk size, overlap, or parsing logic, old embeddings no longer match the new strategy. Rebuild the index instead of mixing old and new chunks.

Tables and lists need special handling
A table row without its headers is often meaningless. Keep headers with table chunks or transform tables into readable text.

Too much overlap creates duplicates
Overlap helps at boundaries, but large overlap can make search results repetitive and waste context.

Chunking cannot fix bad source content
If the document is outdated, contradictory, or vague, the RAG system will retrieve outdated, contradictory, or vague text. Clean the content as well as the index.