Comparison
Preference tuning (RLHF, DPO)vsSupervised fine-tuning (SFT)
Preference tuning (RLHF, DPO)
you train on pairs where one answer is better than the other, rather than on a single correct target.
Training on comparisons ('this answer is better than that one') rather than single targets. RLHF does it via a learned reward model and reinforcement learning; DPO trains directly on the preference pairs and is far simpler to run. This stage is much of what makes chat models feel helpful.
Full entry →Supervised fine-tuning (SFT)
you hand it a few hundred prompt-and-ideal-answer pairs, which is what people mean by fine-tuning when they do not qualify it.
Fine-tuning on prompt-and-ideal-response pairs, the bread-and-butter phase for teaching format and behaviour. When someone says fine-tuning without qualification, they usually mean this.
Full entry →