RLHF 101: A Technical Tutorial

Summary

A technical tutorial from CMU's ML blog covering the foundations and mechanics of Reinforcement Learning from Human Feedback (RLHF), including reward modeling, policy optimization, and alignment objectives. It provides an accessible yet rigorous introduction to how RLHF is used to align large language models with human preferences. The tutorial bridges theory and practice for researchers and practitioners entering the field.

Key Points

• •Covers the core RLHF pipeline: supervised fine-tuning, reward model training from human comparisons, and RL-based policy optimization (e.g., PPO).
• •Explains how human preference data is collected and used to train a reward model that proxies human judgment.
• •Discusses key challenges in RLHF including reward hacking, overoptimization, and distribution shift between the reference and trained policy.
• •Provides technical grounding in the KL-divergence penalty used to prevent the policy from deviating too far from the base model.
• •Serves as an educational reference for understanding why RLHF has become a dominant alignment technique in modern LLM development.