Chunking Support Content That Actually Retrieves

Retrieval quality is mostly decided before any model is involved. It is decided when you cut the source material into pieces, because a piece is the unit that gets retrieved, and no amount of prompt engineering recovers an answer that was split down the middle.

Here is what we have learned cutting up other people's websites and documents.

The size argument is really a precision argument

Small chunks retrieve precisely and arrive without context. Large chunks carry context and dilute the thing you were matching on.

A 200-token chunk containing exactly the refund window matches a question about the refund window beautifully. It may also have lost the sentence three paragraphs earlier establishing that this section is about wholesale orders. The model gets a precise, confident, wrong answer.

A 2000-token chunk keeps that qualifier and buries the refund window among nine other policies, so the embedding is an average of ten topics and matches every one of them weakly.

Neither end is correct in general. What resolves it is cutting on meaning rather than on length: a chunk should be the smallest span that is still true on its own. That is usually a section under a heading, which is why headings in the source matter more than any parameter you can tune.

Navigation is in every chunk and it is invisible

The single most damaging thing about crawling a real website is that every page carries the same furniture. Header, nav, footer, cookie banner, newsletter form.

Naively chunked, that boilerplate lands in every chunk from every page. Now every chunk shares a substantial block of identical text, embeddings across the whole corpus drift toward each other, and retrieval gets flatter and less discriminating the more pages you index.

It fails in a way that does not look like a bug. Nothing errors. Results are simply mediocre, uniformly, and the natural response is to blame the embedding model.

Strip the furniture before chunking. If you index a site and retrieval gets worse as coverage grows, this is almost always why.

PDFs are not documents

A PDF is a description of marks on a page. It has no concept of a paragraph, and it certainly has no concept of a table.

Two-column layouts extract as interleaved lines from both columns. Tables extract as a stream of cells whose relationship to their headers is gone, which means a specification table, the single most useful object in a product manual, becomes a list of numbers attached to nothing.

Where tables carry real answers, they need converting to a structured form before chunking, and each row usually needs to carry its headers. A row that reads Range | 456 km is retrievable. A row that reads 456 is noise.

Overlap is a patch, not a fix

The standard remedy for boundary problems is overlapping chunks by a couple of hundred tokens, so a fact cut in half survives whole in one of the two.

It works and it costs. Every overlapped span is stored and embedded twice, and duplicate content in the index means retrieving the same passage in two positions of a top-k result, spending the budget you had for a second perspective on a copy of the first.

Modest overlap at semantic boundaries is worth it. Heavy overlap as a substitute for good boundaries is paying storage to paper over a chunking strategy that should have been fixed.

The index bug that cost us weeks

Everything above is chunking. This one is retrieval, and it is the most expensive thing we have hit in this area.

We ran an IVFFlat index over our vector table. Collections were stored together, filtered at query time. Ask for the top 10 results in a collection and you would get 10. Usually.

Sometimes you would get 4. No error, no warning, no signal of any kind.

IVFFlat is an approximate index: it searches a subset of clusters rather than the whole table. When the index spans multiple collections, the clusters it probes are chosen across all of them, so the rows it returns get filtered down to your collection afterward. If your collection is a minority of the table, the probed set contains few of your rows, and you receive whatever survives the filter.

The failure is silent and it is worst on your smallest collections, which are usually your newest ones. The agent answers using four chunks when it asked for ten, and it answers confidently, because a short context looks exactly like a complete one from the inside.

The fix was to stop letting one approximate index span collections that get filtered independently. The lesson generalises: an approximate index plus a post-filter silently returns fewer results than you asked for, and nothing in your stack will tell you.

If your retrieval is occasionally thin and you cannot reproduce it, count the rows you actually got back rather than the rows you requested. We did not, for a long time.

How to find these

All of these failures share a property: the system does not report them. Boilerplate contamination, mangled tables, split facts, short result sets. None throws.

The only reliable detection is to look at what was actually retrieved for a real question, as text, with your own eyes. Not the answer, the chunks. Do that twenty times against real questions and you will find every problem in this post inside an afternoon.

Retrieval quality is not a metric you can watch. It is a thing you have to read.