Chunking is the least glamorous decision in a RAG pipeline and one of the most consequential. Get it wrong and no amount of prompt engineering or model upgrade fixes the underlying problem: the retriever keeps handing the model incomplete or context-free fragments, and the model can only be as good as what it’s given to work with.
Why Chunking Determines Retrieval Quality More Than Most Teams Expect
A common assumption is that retrieval quality is mostly about embedding model choice or vector database performance. In practice, chunk boundaries frequently matter more — an excellent embedding model applied to a badly-cut chunk still embeds the badly-cut chunk. If a chunk splits a table in half, cuts off a definition right before the sentence that explains it, or bundles three unrelated topics into one block because it hit a fixed character count, no downstream component can fully recover the lost or diluted meaning. Chunking is where a document’s structure either gets preserved or destroyed before anything else in the pipeline gets a chance to work with it.
Chunk Size: The Core Tradeoff
| Chunk Size | Retrieval Precision | Context Completeness | Typical Use Case |
|---|---|---|---|
| Small (~128–256 tokens) | High — retrieves very targeted, specific passages | Low — often missing surrounding context needed to fully understand the passage | Fact lookup, FAQ-style Q&A, precise citation needs |
| Medium (~256–512 tokens) | Balanced | Balanced | General-purpose document Q&A — the most common starting point |
| Large (~512–1024+ tokens) | Lower — retrieval becomes less precise as chunks cover more ground | High — preserves more surrounding context and nuance | Complex reasoning tasks, technical or legal documents where context loss is costly |
There’s no universally correct size — it’s a direct tradeoff between precision (retrieving exactly the relevant passage) and completeness (retrieving enough surrounding context that the passage is actually usable). Teams that pick a chunk size once and never revisit it are usually leaving retrieval quality on the table, since the right size depends heavily on document type and query pattern, not a general best practice number.
Overlap: Preventing the Boundary Problem
Fixed-size chunking without overlap creates a specific, easy-to-miss failure: a critical sentence sits exactly on the boundary between two chunks, so neither chunk contains the complete idea, and neither one retrieves well for a query about that specific point. Overlap — typically 10–20% of the chunk size — duplicates a portion of content between adjacent chunks specifically to prevent this boundary-splitting problem. The cost is straightforward: more total chunks, more storage, and marginally more redundant content in retrieval results, which is a reasonable trade for not silently losing information at every chunk boundary.
Fixed-Size vs Semantic Chunking
Fixed-size chunking (splitting by a set token or character count) is simple to implement and predictable, but it’s blind to document structure — it will cut mid-sentence, mid-table, or mid-list item exactly as often as it lands on a clean boundary, purely by coincidence. Semantic chunking instead splits along structural or meaning-based boundaries — paragraphs, sections, or points where the topic actually shifts — which better preserves coherent ideas per chunk at the cost of more complex implementation and variable chunk sizes that are harder to reason about for downstream token-budget planning.
A practical middle ground many production systems use: structure-aware chunking that respects document formatting (never splitting inside a table, code block, or list) while still targeting a roughly consistent size range — capturing most of semantic chunking’s benefit without the full complexity of true semantic boundary detection.
Document-Type-Specific Considerations
- Tables and structured data: should generally be kept whole as a single chunk, or chunked row-group by row-group with headers repeated in each chunk — splitting a table’s header from its data rows makes both halves nearly useless for retrieval.
- Code documentation: function or class boundaries make much better chunk boundaries than arbitrary token counts, since a partial function definition retrieved alone is rarely useful.
- Long-form technical or legal text: section and subsection headers are usually the best chunk boundaries, since these documents are already structured by the author into meaningful, self-contained units.
- Conversational or transcript data: chunking by speaker turn or topic shift (rather than fixed length) tends to preserve the back-and-forth context that’s often essential to understanding any single exchange.
How to Actually Tune This Instead of Guessing
Chunking strategy should be evaluated against retrieval quality metrics on a representative test set of real queries — not chosen once from a blog post’s recommended default and left alone. A practical evaluation loop: build a set of realistic queries with known correct source passages, test multiple chunk size/overlap combinations against that set, and measure retrieval precision and recall directly rather than judging by how the final generated answer “feels.” Chunking decisions made this way, against your actual document types and query patterns, consistently outperform generic defaults borrowed from an unrelated use case.
