RLHF — Aligning LLMs with Human Feedback

A raw pretrained LLM (GPT-3, LLaMA-2 base) is trained to predict next tokens — not to be helpful, harmless, or honest. It may complete harmful requests, hallucinate confidently, or give rambling non-answers. RLHF transforms a text predictor into an aligned assistant.

Human labelers write ideal (prompt, response) pairs across diverse topics: • Factual QA, coding, reasoning, creative writing, safety refusals The base LLM is fine-tuned on these demonstrations. This gives the model the right behavioral format but relies on labeler quality and coverage.

For each prompt, the SFT model generates K responses (e.g., K=4). Human raters rank the responses from best to worst. A reward model (another LLM with a scalar head) is trained to predict human preference scores from these comparisons: Loss = −E[log σ(r(x, y_w) − r(x, y_l))] Where y_w = preferred response, y_l = less preferred response.

The SFT model (now the policy π) generates responses. The RM scores them. PPO (Proximal Policy Optimization) updates π to maximize reward: Objective = E[r(x, y)] − β · KL(π || π_SFT) The KL penalty prevents the policy from 'reward hacking' — generating nonsense that scores high but isn't actually good. β controls alignment vs reward tradeoff.

DPO (2023) skips the reward model entirely. It directly optimizes the policy on preference pairs: L_DPO = −E[log σ(β·log(π/π_ref)(y_w|x) − β·log(π/π_ref)(y_l|x))] Simpler, more stable, same or better results than PPO. Now the dominant approach for open-source model alignment (Zephyr, Tulu, OpenHermes).