← Back to blog

RAG Chunking Strategy: How to Split Documents for Better Answers

November 15, 2026
RAG Chunking Strategy: How to Split Documents for Better Answers

The best chunking strategy for most first versions is structure-aware splitting: cut documents along their own headings, clauses and paragraphs, keep chunks at roughly 256 to 512 tokens, and attach metadata to every piece. Then measure it against real questions from your users. A chunk is one piece of a document, and a RAG system (retrieval-augmented generation, where an AI model answers from your documents) can only answer from the pieces it finds. If the cut is wrong, the answer is wrong. This is the layer I check first when a founder's prototype gives vague or incorrect answers, and it is usually cheaper to fix than it looks.

Hanad Kubat
Build a Working AI Product
Turn a promising prototype into working software, with a fixed scope and price and code you own from the first commit.
Visit

Key Takeaways

  • Chunking decides what the AI model gets to read. Pieces that are too big waste money and bury the answer. Pieces that are too small lose the context around it.
  • Start with structure-aware or recursive splitting at 256 to 512 tokens, with modest overlap. Fancier methods rarely win on their own.
  • Match the method to the document: contracts by clause, manuals by step, FAQs by question and answer, reports with their tables kept intact.
  • Metadata (source, date, section, access rights) matters as much as the split itself, and it is what makes a pipeline maintainable.
  • There is no universal best setting. A small set of real questions with known answers is how you pick one.

Table of Contents

What chunking is and why it decides answer quality

An AI model cannot read your whole document library every time someone asks a question. So the system splits each document into pieces, converts each piece into a list of numbers called an embedding (a numerical fingerprint of its meaning), and stores them in a vector database. When a question arrives, the system finds the pieces whose fingerprints are closest to the question and hands those to the model. MongoDB describes this flow: chunk, embed, store, retrieve by similarity, optionally rerank, then generate.

RAG ingestion pipeline from chunking to answers

Microsoft's architecture guidance defines the goal as chunks that each hold semantically relevant content. Oversized chunks are expensive and can overwhelm token limits. Chunks that are too small miss context that spans several pieces.

Chroma's documentation gives a concrete failure. Someone asks how long the connection timeout is. The system retrieves the piece that names the timeout setting but not the piece that says "30 seconds", because the answer and its context were split across two chunks. The model never sees the number, so it guesses or hedges.

The same documentation describes the chain reaction. Retrieval returns partial matches. Engineers compensate by retrieving more pieces per question, which raises token costs and dilutes the prompt. The model then produces hedged or wrong answers because it can only work with what it is given. If your prototype "hallucinates", look here before blaming the model. I cover the wider picture in reducing AI hallucinations in production.

Why a prototype's default chunking stops being good enough

AI app builders and no-code tools usually wire up a document feature with one fixed setting for everything. That is a reasonable start and it works in a demo with three clean PDFs. It stops working when real documents arrive: scanned contracts, spreadsheets pasted into Word, 80-page manuals with nested headings, price tables.

The symptoms are consistent. Answers are almost right. The right document is cited but the wrong paragraph is quoted. A number is correct in one answer and missing in the next. Costs climb because the system retrieves large amounts of text to compensate. None of this is a model problem, and swapping to a bigger model rarely fixes it.

The chunking strategies, compared

Databricks groups the options into fixed-size, paragraph-based, format-specific and semantic chunking. Here is how they differ in practice.

StrategyHow it splitsWorks well forWatch out for
Fixed-sizeEvery N tokens, with a small overlapQuick first pass, uniform textCuts mid-sentence and ignores meaning
RecursiveParagraphs first, then lines, then sentences, until the piece fitsGeneral default for mixed textCannot detect a topic change inside one paragraph
Sentence-basedOne or a few sentences per chunkTranscripts, short Q&AA single sentence often lacks context
Structure-awareFollows headings, clauses, list items, Markdown or HTMLContracts, manuals, policies, reportsNeeds consistent formatting in the source
SemanticStarts a new chunk when the topic shifts, measured by embeddingsLong flowing proseCan over-fragment; more complex to run
ContextualAdds a short LLM-written summary to each chunkDocuments where a piece makes no sense aloneAn extra model call per chunk, redone when sources change

A few findings are worth knowing before you get attached to a clever method.

  • A 2026 academic evaluation, Beyond Chunk-Then-Embed, found that the best strategy depends on the task, and that simple structure-based methods outperformed LLM-guided ones for retrieval across a whole collection of documents.
  • A February 2026 evaluation on ten SEC filings and 42 questions found structure-aware chunking reached 0.877 context recall, ahead of fixed-size (0.826), recursive (0.822) and semantic (0.759). Fewer chunks beat more chunks because the structure was preserved.
  • A University of Twente study on medical guideline documents found semantic and propositional chunking scored lowest, while sliding-window chunking at 256 and 512 tokens scored highest.

The pattern: boring and structure-respecting tends to beat clever. That fits how I like to build generally, and it is why I would start there for your first version.

Chunk size and overlap: where to start

Tokens are the units AI models count text in, roughly three-quarters of an English word each. Chunk size is how many tokens go in one piece. Overlap is how many tokens two neighboring pieces share, so a sentence cut at a boundary still appears whole in one of them.

The trade-off is stated plainly in the vendor documentation. Databricks notes that smaller chunks focus on detail but lose surrounding context, while larger chunks capture more context but can include irrelevant material or cost more to process. Cohere's cookbook adds that overlap works as a buffer for information split at a boundary, at the cost of redundancy.

The Twente study shows the trade-off in numbers. In its tests, 1024-token chunks scored highest on contextual recall, because a bigger piece is more likely to contain the answer. They scored lower on precision because they carry more noise. 256-token chunks showed the opposite pattern, with higher precision (0.729) but lower content overlap.

So what should you pick?

  1. Start at 256 to 512 tokens. Unstructured's guidance suggests about 250 tokens, roughly 1,000 characters as a baseline to experiment from. The SEC filing evaluation above found 512 worked best for those documents.
  2. Use modest overlap, around 10 to 20 percent. One analysis of a Databricks financial-document sweep found that raising overlap from 20 percent to 50 percent added less than a point of correctness and roughly doubled the index size. Heavy overlap fills your database with near-duplicates and returns the same passage twice.
  3. Count in tokens, not characters. Microsoft recommends token-based sizing because tokens follow meaningful units of language, and it points to a chunk visualizer so you can see how real documents split.
  4. Do not treat any number as final. The same Databricks sweep found correctness rising with chunk size on financial documents, while the Twente medical study found 1024 worst overall. Different documents, different answers. Databricks itself calls the process iterative and context-dependent.

How to chunk by document type

This is where non-technical founders can contribute the most, because you know your documents. One handbook on data engineering for AI recommends chunking by the document's own logic:

  • Policies and contracts: by chapter and clause. A clause is a complete thought, and splitting it destroys the exception that follows the rule.
  • Product manuals and procedures: by feature and operating step, so a step is never separated from its warning.
  • FAQs: as question-answer pairs, with tags, never split between question and answer.
  • Reports and invoices: keep a table together with its title, units, notes and the paragraph that explains it. A table cut in half is worse than no table.

That handbook cites research finding that element-type chunking beat fixed-length chunking for financial reports. It also argues overlap should be chosen per document type rather than applied everywhere.

If you run a clinic, law practice, trade business or logistics operation, your documents are probably consistent in form: the same intake form, the same contract template, the same delivery note. That is good news. Consistent structure is exactly what structure-aware splitting needs, and it often means you can skip the more expensive methods.

Metadata: the part founders skip

Every chunk should carry information about where it came from. At minimum: the source document, its location in that document, the date it was last modified, the language, and who is allowed to see it. One engineering handbook lists a fuller minimum metadata schema including source, URI, author, modified date, a content hash, chunk position, tenant, access tags and the embedding model version.

In plain terms, that metadata lets you do three things you will need within a few months:

  • Show citations, so users can click through to the original page and trust the answer.
  • Filter by permission, so one customer's documents never appear in another customer's answers. If you sell to businesses, tenant isolation is not optional.
  • Update without rebuilding. The same handbook warns that without a freshness loop, stale embeddings make the system confidently cite outdated content. A content hash tells you which documents changed, so you re-process only those.

Contextual retrieval: when a chunk makes no sense alone

Some chunks are useless on their own. "Revenue grew 3 percent over the previous quarter" does not say which company or which quarter. Contextual retrieval fixes this by having a language model write a short note, typically 50 to 100 tokens, and prepending it to each chunk before indexing. The method comes from Anthropic, and a summary of its results reports 49 percent fewer failed retrievals, rising to 67 percent when combined with a reranker (a second model that re-sorts the retrieved pieces).

The cost is one model call per chunk across your corpus, repeated when the source changes. For many first versions that is an upgrade for later, after the basic split and metadata are solid and you can see which questions still fail.

Where chunking sits in the ingestion pipeline

Ingestion is everything that happens before a user asks anything. Chroma describes the common sequence: split the data into chunks, collect metadata to attach to each, then insert the records into a collection where they are embedded.

In practice a production pipeline has more steps around that core:

  1. Extract. Pull clean text out of PDFs, Word files, emails or web pages. Scans need OCR. Tables need special care. Bad extraction ruins every later step, and it is the most underestimated part. My piece on shipping document search with extraction first goes deeper.
  2. Clean. Remove headers, footers, page numbers and boilerplate that repeat on every page.
  3. Chunk. Apply the strategy for that document type.
  4. Enrich. Attach metadata, and contextual notes if you use them.
  5. Embed and store. Record the embedding model version so you can re-index cleanly if you change models.
  6. Refresh. Re-process only what changed.

For a wider look at planning RAG for production, this overview of engineering-first RAG architecture is a good companion.

How to choose and check your chunking, step by step

Here is the routine I follow, and where each step usually goes wrong.

  1. Collect 50 to 150 real questions with known answers. Ask your staff or early users. The common mistake is writing questions yourself that mirror the document's wording, which flatters every setting. The method is in my RAG evaluation playbook.
  2. Set a baseline. Run recursive splitting at 512 tokens with light overlap and record how often the correct chunk appears in the top results.
  3. Change one thing at a time. Chunk size first, then method, then overlap. Changing three settings at once tells you nothing.
  4. Read the failures. Chroma's guidance is useful here: if the right chunks are not retrieved at all, try smaller chunks with more overlap; if they are retrieved but rank low, add context to the chunks and use metadata filters.
  5. Move to structure-aware splitting if your documents have consistent headings or clauses. This is usually the biggest single gain.
  6. Add contextual notes or reranking last, only for the questions that still fail.
  7. Re-run the same questions after every change, including model swaps and new document types, so a fix in one place does not silently break another. The metrics worth tracking are covered in AI evaluation metrics for production.

My view: why I start boring

I get called in when a prototype has stalled: the demo worked, then real documents arrived and answers went soft. In almost every case the fix is not a cleverer model. It is a clean extraction step, a split that follows the document's own structure, metadata on every piece, and a small set of real questions that proves the change helped.

I build this as one person, end to end, so the engineer you talk to is the engineer writing the pipeline. You own the code from the first commit, and I keep the stack deliberately boring, TypeScript, Node, SQL and Cloudflare Workers, so the next developer can read it without me. Fixed price and fixed scope also mean you know the cost of the ingestion pipeline before it starts, not after the third change request.

If an off-the-shelf tool fits your documents, I will tell you so. If it does not, the chunking choices above are the first things I would settle with you, using your documents and your questions rather than a blog's default numbers.

Hanad Kubat

Getting a working version built

If you have a prototype that retrieves the wrong passages, or documents you want to make searchable and have no code yet, I build the first working version as a fixed-price engagement, in two to four weeks, with the ingestion pipeline, retrieval and evaluation set included in scope. Builds start from €12,000. You see working progress before you pay, and the code is yours from the first commit. Details are on my site.

FAQ

What is the best chunk size for RAG?

There is no single best size, but 256 to 512 tokens is a sound starting range for most text. Published evaluations disagree on the winner because documents differ: one medical study found 256 and 512 best and 1024 worst, while a financial-document sweep saw correctness rise with size. Test two or three sizes against your own questions.

How much overlap should I use?

Start with 10 to 20 percent of the chunk size. Overlap protects sentences cut at a boundary, but heavy overlap adds little accuracy and can double the index size with near-duplicate pieces. Structure-aware splitting reduces the need for overlap because the cuts fall at natural boundaries.

Should I use semantic chunking?

Not as a first choice. It splits by topic shift and can produce many tiny fragments, and in the evaluations above it scored below simple structure-aware and recursive methods. Try it after your baseline, on long flowing prose where headings are missing.

Do I need to re-chunk when I change the embedding model?

You need to re-embed every chunk, because vectors from different models are not comparable. Re-chunking is only needed if your size or method changes. Storing the embedding model version in each chunk's metadata makes the switch a controlled job rather than a guess.

How do I know my chunking is working?

Run a fixed set of real questions with known answers and measure how often the correct chunk is retrieved. Repeat after every change. Without that set, you are judging by a few lucky demos.

Sources