Understanding RLHF

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the technique that transformed large language models from text predictors into helpful, harmless assistants. It is the key ingredient behind ChatGPT, Claude, and other aligned LLMs.

The Three Stages of RLHF

Stage 1: Supervised Fine-Tuning (SFT)

Start with a pre-trained LLM and fine-tune it on high-quality demonstration data:

$\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x,y) \sim \mathcal{D}}\left[\log \pi_\theta(y | x)\right]$

where $(x, y)$ are prompt-response pairs curated by human annotators.

Stage 2: Reward Model Training

Train a reward model $r_\phi(x, y)$ from human preference data. Given pairs $(y_w, y_l)$ where $y_w$ is preferred over $y_l$:

$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))\right]$

This follows the Bradley-Terry model of preferences.

Stage 3: RL Optimization (PPO)

Optimize the policy against the reward model using Proximal Policy Optimization:

$\mathcal{L}_{\text{PPO}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta}\left[r_\phi(x, y) - \beta \cdot D_{KL}(\pi_\theta \| \pi_{\text{ref}})\right]$

The KL penalty $\beta \cdot D_{KL}$ prevents the model from deviating too far from the SFT model and exploiting the reward model.

DPO: A Simpler Alternative

Direct Preference Optimization (DPO) eliminates the need for a separate reward model:

$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]$

DPO shows that the optimal policy under the RLHF objective has a closed-form solution, allowing direct optimization from preferences.

Challenges

• Reward hacking: The model may find shortcuts to maximize the reward without being genuinely helpful
• Annotation quality: Human preferences are noisy and inconsistent
• Distribution shift: The policy moves away from the training distribution during RL
• Scalable oversight: As models become more capable, human evaluation becomes harder

Conclusion

RLHF (and its variants like DPO) bridges the gap between raw language modeling capability and aligned, useful AI behavior. It remains an active area of research in AI alignment.

References

1. Ouyang et al., "Training language models to follow instructions with human feedback," NeurIPS 2022.
2. Rafailov et al., "Direct Preference Optimization," NeurIPS 2023.
3. Christiano et al., "Deep RL from Human Preferences," NeurIPS 2017.