"Chunking is a product decision, not a preprocessing step"

Ask your document assistant a question whose answer is a whole section, and watch it respond with a fragment. Ask it for one specific figure, and watch it drown the figure in three paragraphs of adjacent noise. Both failures trace back to the same line of code: the one where somebody set a chunk size to 512 tokens because that was the default, and never thought about it again.

Chunking gets treated as preprocessing, a mechanical step between loading documents and embedding them, on a par with stripping whitespace. It is nothing of the sort. The chunker decides what the model is allowed to see. Every answer your system will ever give is assembled from the pieces the chunker produced, and no amount of prompt engineering downstream can reunite information the chunker separated. That makes it a product decision wearing a preprocessing costume.

The core trade-off is straightforward to state and awkward to resolve. Small chunks retrieve precisely: the embedding represents one idea, similarity search finds it cleanly, and the model receives a dense, relevant context. But small chunks amputate context, the clause without its exception, the step without the warning above it. Large chunks preserve context and retrieve mushily: the embedding becomes an average of five topics, similar to many queries and a strong match for none. There is no universally correct point on this dial, because the right setting depends on what your users ask.

That is the product question hiding inside the technical one. A support tool answering "what is the refund window?" wants small, precise chunks, the answer is a sentence. A tool asked "summarise our obligations under this contract" wants large ones, or the summary will be built from disconnected shards. Most real systems face both kinds of question, which is why teams end up with parent-child schemes, retrieve against small chunks for precision, hand the model the surrounding section for context, or multiple granularities in the same index. That machinery is not gold-plating; it is what taking both question types seriously looks like.

Before touching size at all, fix boundaries, because where you cut matters more than how much. Fixed-length splitting slices mid-sentence and mid-thought as a matter of course. Structure-aware splitting, on headings, paragraphs, list items, section breaks, keeps units of meaning intact, and documents are full of structure that says so: a heading announces a topic; a numbered list is one procedure, not seven facts. Throwing that signal away and cutting every 512 tokens regardless is the single most common self-inflicted wound I see in RAG systems.

Tables deserve their own paragraph, because naive chunking quietly destroys them. Split a table across chunks and you separate headers from rows; the model then pairs numbers with the wrong labels, and the resulting answer is worse than none, it is wrong with a citation. Tables want to be kept whole where possible, carried with their headers when they must be split, and often summarised into prose alongside the raw grid so that semantic search has something to grip. Code blocks have the same character: half a function is not a smaller amount of useful context, it is zero useful context.

Attach context while you still can. At chunking time you know the source document, the section heading, the page, the version date. Stamp all of it onto every chunk, it costs almost nothing during ingestion and buys you filtered retrieval, honest citations, and the ability to expire stale content, none of which can be reconstructed later from a bare string of text. The published evidence here is unusually clean: Anthropic's contextual retrieval work (https://www.anthropic.com/news/contextual-retrieval) found that prepending a short, document-aware description to each chunk before embedding cut retrieval failures by 35 per cent, and by 49 per cent when combined with keyword search. Nothing about the model changed. Only what the chunks carried.

The uncomfortable part: you cannot reason your way to the right strategy from an armchair. You have to measure. Take thirty real questions with known source passages, run them against candidate chunking schemes, and count how often the passage that contains the answer actually comes back in the top results. It is an afternoon of work, and I have watched that afternoon change retrieval hit-rate by double digits, a bigger improvement than a model upgrade, at a fraction of the cost. Re-chunking an index is cheap; discovering in production that your chunks cannot answer your users' questions is not.

So when someone on the team asks "what chunk size are we using?", the honest answer starts with "what are our users asking?", and if nobody knows, that is the real gap. This is exactly the work I do: making retrieval systems answer the questions they were actually built for. Get in touch via the contact form.