RLHF takes a model that already predicts plausible text and shapes its behaviour toward what people want. Human reviewers rank several candidate outputs for the same prompt; those rankings train a separate reward model, which then guides further training on the original model so it produces more of what raters preferred and less of what they didn’t. It’s the step that turns a raw next-token predictor into something that behaves like an assistant.
RLHF is why today’s frontier chat models are dramatically more usable and less prone to obviously harmful output than their pre-RLHF base versions. It is also, by design, an empirical fix rather than a formal one: it optimises for what human raters rewarded in the examples they saw, not for a mathematically specified notion of safety that would hold outside those examples. A model can learn to satisfy the letter of what raters rewarded while missing the underlying intent, a failure mode researchers call reward hacking.
Treat RLHF as safety engineering under uncertainty, the same category as red-teaming and capability thresholds, not as a guarantee. It reliably reduces the rate of bad behaviour on cases resembling its training examples; it has never been shown to guarantee good behaviour on cases that don’t.