Reinforcement Learning for LLMs: Complete Guide to RLHF, DPO, and GRPO

Contents

Three acronyms, one job: turning a base model into an assistant

A pretrained language model can finish your sentence. It can't reliably follow your instructions, refuse harmful requests, or solve a hard math problem step by step. Post-training closes that gap, and reinforcement learning sits at the heart of it.

Three acronyms dominate the conversation: RLHF, DPO, and GRPO. People argue about them like sports fans, and the arguments confuse more than they explain. This guide explains what each method does, how they differ, when to use each one, and where they break — the training-side companion to the evaluation guides on this site. This field moves fast, so the methods here combine core-paper knowledge with 2026 practitioner write-ups: check the original papers and current library docs before you build anything.

The Big Picture: From Base Model to Aligned Model

Most modern post-training pipelines follow the same outline:

1. Pretraining: the model learns language from huge text corpora.
2. Supervised fine-tuning (SFT): the model learns to follow instructions from curated examples.
3. Preference or reward optimization: the model improves further using feedback, either human preferences or automatic checks.

SFT shows the model what good answers look like. Reinforcement learning teaches it to prefer better answers over worse ones, even when no single "correct" answer exists. That second step is where RLHF, DPO, and GRPO live.

RLHF: The Original Recipe

Reinforcement Learning from Human Feedback powered the first wave of aligned chat models. It works in three stages after SFT.

Stage 1: Collect Preferences

Human raters see two responses to the same prompt and pick the better one. You gather thousands of these comparisons.

Stage 2: Train a Reward Model

A separate model learns to predict which response humans prefer. It outputs a scalar score for any prompt and response. Training uses a pairwise loss, which pushes the winning response's score above the losing response's score.

Stage 3: Optimize the Policy With PPO

The language model (the "policy") generates responses, the reward model scores them, and Proximal Policy Optimization (PPO) updates the policy to earn higher scores. A KL penalty keeps the policy close to a frozen reference model, so it doesn't drift into gibberish that games the reward model.

What RLHF Does Well and Badly

• Strength: it's flexible and well proven, and it learns from online samples as the policy improves.
• Weakness: it's heavy. You juggle up to four models at once (policy, reference, reward model, and a value model), and PPO has many hyperparameters that need careful tuning.
• Weakness: the policy can hack the reward model, finding outputs that score high without being good.

DPO: RLHF Without the RL Loop

Direct Preference Optimization showed that you can skip the reward model and the PPO loop. The key insight: the RLHF objective has a closed-form optimal policy, so you can optimize directly on preference pairs with a simple classification-style loss.

How DPO Works

You need a dataset of prompts, each with a chosen and a rejected response. DPO increases the probability of the chosen response relative to the rejected one, measured against a frozen reference model. A parameter called beta controls how far the policy may drift from the reference:

import torch.nn.functional as F

def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    """Inputs: summed log-probabilities of each full response."""
    logits = beta * ((pi_chosen - ref_chosen) - (pi_rejected - ref_rejected))
    return -F.logsigmoid(logits).mean()

What DPO Does Well and Badly

• Strength: it's simple, stable, and cheap. You train with two models (policy and reference) and no sampling loop.
• Strength: it's well understood theoretically, with a clear link to KL-regularized RLHF.
• Weakness: it's offline. It learns only from the preference pairs in your dataset. If the data doesn't cover a failure mode, DPO can't fix it, and the model can drift away from the data's distribution as training proceeds.
• Weakness: it tends to underperform on reasoning tasks where exploring many attempts matters.

The popular slogan "DPO is RLHF without RL" oversimplifies. DPO optimizes a similar objective with a different mechanism, and the offline limitation matters in practice.

The DPO Family

A crowd of variants followed, each tweaking the loss: IPO, KTO, ORPO, SimPO, and others. KTO, for instance, learns from simple thumbs-up and thumbs-down labels instead of paired comparisons. Try plain DPO first, and explore variants when you hit a specific problem like noisy labels or length bias.

GRPO: Group Relative Policy Optimization

GRPO came out of DeepSeek's math work and became famous through the DeepSeek-R1 reasoning model. It targets a different setting: tasks where you can verify an answer automatically.

The Key Ideas

GRPO generates a group of responses for each prompt, scores each one, and compares them to each other. It computes each response's advantage by comparing its reward to the group average, scaled by the group's spread. That removes the need for a separate value model, which saves memory:

def group_advantages(rewards):
    """rewards: tensor of shape (G,) for one prompt's G responses."""
    return (rewards - rewards.mean()) / (rewards.std() + 1e-6)

The policy then updates with a PPO-style clipped objective plus a KL term, so good responses become more likely and bad ones less likely.

Verifiable Rewards (RLVR)

GRPO shines with Reinforcement Learning with Verifiable Rewards. Instead of a learned reward model, an objective check supplies the reward:

• Math: does the final answer match the known solution?
• Code: do the unit tests pass?
• Format: does the output follow the required structure?

A verifier can't be "talked into" a high score the way a learned reward model can, which reduces one kind of reward hacking.

What GRPO Does Well and Badly

• Strength: it's on-policy, so the model explores and learns from its own attempts. That's why it suits reasoning tasks.
• Strength: it drops the critic, which cuts memory and complexity compared with PPO.
• Weakness: it needs a reliable reward signal. Where correctness isn't verifiable, such as tone or helpfulness, GRPO needs a learned reward model or judge, and the reward hacking problem returns.
• Weakness: it generates many samples per step, so training costs real compute.

The GRPO Family

Several refinements address GRPO's quirks:

• Dr. GRPO: adjusts the normalization to remove biases, such as length effects and difficulty weighting.
• DAPO: adds tricks like "clip-higher" to encourage exploration and dynamic sampling that skips groups where every response gets the same reward, since those groups carry no learning signal.
• RLOO: a close cousin that uses a leave-one-out baseline. TRL ships it alongside GRPO, and the two behave similarly.

A 2026 paper also argues that GRPO connects deeply to DPO, and it reports that a group size of just two can match much larger groups on math tasks with far less training time. Treat that as an interesting single result, not settled advice.

Figure 1: Post-training decides how the model behaves — the acronym you pick changes the machinery, not the goal

Side-by-Side Comparison

• RLHF (PPO): needs a reward model and a value model. Learns online. Heavy and flexible. Best when you have a trusted reward model and the engineering resources.
• DPO: needs preference pairs only. Learns offline. Light and stable. Best for style, helpfulness, and safety tuning with good data.
• GRPO / RLVR: needs a verifier or reward function. Learns online. Moderate cost. Best for reasoning, math, and code.

How to Choose

Walk through these questions:

1. Do you have a strong SFT model? Always start there. RL on a weak base wastes compute.
2. Can you verify answers automatically? If yes, use GRPO with a verifier.
3. Do you have good preference pairs? If yes, and the task is about style, tone, or safety, try DPO first.
4. Does DPO fall short or drift? Move to online DPO or RLHF with PPO.
5. Do you lack human labels? Consider RLAIF, where an AI judge supplies feedback, as in Constitutional AI approaches — calibrated against human labels first, since judges carry measurable biases.
6. How much compute do you have? DPO costs the least, and online methods cost more.

My default advice: SFT first, then DPO for preference tuning, and GRPO for verifiable reasoning. Reach for PPO-based RLHF only when simpler methods fail.

Practical Tooling

Several open-source libraries implement these methods:

• Hugging Face TRL: includes DPO, GRPO, and RLOO trainers. It's the easiest starting point for small to moderate experiments.
• verl and OpenRLHF: aimed at larger-scale training.
• Unsloth: focuses on memory-efficient training on limited hardware.

Here are minimal TRL sketches. Argument names change between TRL versions, so check the current docs:

from trl import DPOConfig, DPOTrainer

# dataset columns: prompt, chosen, rejected
trainer = DPOTrainer(
    model=model,
    args=DPOConfig(output_dir="dpo-out", beta=0.1),
    train_dataset=dataset,
    processing_class=tokenizer,
)
trainer.train()

from trl import GRPOConfig, GRPOTrainer

def correctness_reward(completions, answer, **kwargs):
    return [1.0 if extract_final(c) == a else 0.0
            for c, a in zip(completions, answer)]

trainer = GRPOTrainer(
    model="your-small-base-model",
    reward_funcs=correctness_reward,
    args=GRPOConfig(output_dir="grpo-out", num_generations=8),
    train_dataset=dataset,
)
trainer.train()

extract_final stands in for your own function that pulls the final answer out of a completion.

Common Failure Modes

All three methods share a few traps:

• Reward hacking: the model exploits flaws in the reward signal. Use KL anchors, check samples by hand, and prefer verifiable rewards where you can.
• Length bias: preference data often favors longer answers, and models learn to ramble. Control for length in data and evaluation.
• Mode collapse: outputs lose diversity as training narrows the policy.
• Format gaming: the model learns to satisfy a format check without solving the task. Test with held-out problems.
• Capability regression: alignment training can hurt other skills. Run broad evals before and after.
• Weak SFT foundations: RL amplifies what the base model already almost does. It rarely creates skills from nothing.

Evaluate What You Train

Training loss tells you little about quality. Check results with:

• Held-out preference or verifier-based tests that the model never trained on.
• Pairwise comparisons between the old and new model, using human reviewers or a calibrated judge.
• Regression suites that cover general skills and safety behavior — the CI habits from DeepEval apply directly.
• Manual sample reading, since numbers hide weird behavior.

Common Mistakes

• Skipping SFT — RL on a weak base goes nowhere.
• Using DPO for everything — reasoning tasks usually need online exploration.
• Trusting a learned reward model blindly — reward hacking happens fast.
• Ignoring data quality — noisy preferences produce noisy models.
• Treating "GRPO" as one thing — the variants differ in meaningful ways.
• Evaluating only on the training distribution — hold out real test cases.
• Skipping the verifier audit — a reward function with loopholes teaches the model to exploit them.
• Judging by training curves — held-out evals and human labels decide quality.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between RLHF, DPO, and GRPO?

They solve related post-training problems with different mechanisms. RLHF (with PPO) works in three stages after SFT: humans pick the better of two responses, a separate reward model learns to predict those preferences with a pairwise loss that pushes the winning response's score above the loser's, and PPO updates the policy to earn higher reward-model scores while a KL penalty keeps it close to a frozen reference so it doesn't drift into gibberish that games the reward — flexible and proven, but heavy: up to four models at once (policy, reference, reward model, value model), many hyperparameters, and reward hacking. DPO skips the reward model and the RL loop entirely: the RLHF objective has a closed-form optimal policy, so you optimize directly on preference pairs with a classification-style loss that increases the chosen response's probability relative to the rejected one against a frozen reference, with beta controlling drift — simple, stable, cheap (two models, no sampling loop), but offline: it learns only from the pairs in your dataset, can't fix uncovered failure modes, and tends to underperform on reasoning tasks where exploring many attempts matters. GRPO (from DeepSeek's math work, famous via DeepSeek-R1) targets verifiable tasks: generate a group of responses per prompt, score each, and compute advantage relative to the group mean and spread — removing the value model — then update with a PPO-style clipped objective plus KL; on-policy exploration suits reasoning, but it needs a reliable reward signal and many samples per step.

When should you use GRPO with verifiable rewards?

Use GRPO with Reinforcement Learning with Verifiable Rewards when correctness is automatically checkable: math (does the final answer match the known solution), code (do the unit tests pass), and format (does the output follow the required structure). An objective verifier can't be talked into a high score the way a learned reward model can, which removes one kind of reward hacking, and GRPO is on-policy, so the model explores and learns from its own attempts — that's why it suits reasoning tasks. It also drops the critic, cutting memory and complexity versus PPO. The limits: where correctness isn't verifiable — tone, helpfulness, safety — GRPO needs a learned reward model or judge and the reward-hacking problem returns; and generating many samples per step costs real compute. Variants address its quirks: Dr. GRPO adjusts normalization to remove biases like length effects and difficulty weighting; DAPO adds tricks like clip-higher to encourage exploration and dynamic sampling that skips all-same-reward groups, which carry no learning signal; RLOO is a close cousin using a leave-one-out baseline, shipped alongside GRPO in TRL with similar behavior. One 2026 paper argues GRPO connects deeply to DPO and reports that a group size of two can match much larger groups on math tasks with far less training time — an interesting single result, not settled advice.

How do you choose between RLHF, DPO, and GRPO?

Walk through six questions. Do you have a strong SFT model? Always start there — RL on a weak base wastes compute, since RL amplifies what the base model already almost does and rarely creates skills from nothing. Can you verify answers automatically? If yes, use GRPO with a verifier. Do you have good preference pairs, and is the task about style, tone, or safety? Try DPO first. Does DPO fall short or drift? Move to online DPO or RLHF with PPO. Do you lack human labels? Consider RLAIF, where an AI judge supplies feedback, as in Constitutional AI approaches — calibrated against human labels first, since judges carry biases. How much compute do you have? DPO costs the least; online methods cost more. The default advice: SFT first, then DPO for preference tuning, and GRPO for verifiable reasoning; reach for PPO-based RLHF only when simpler methods fail. Data quality decides outcomes across every path: noisy preferences produce noisy models, so test your verifier for loopholes before training, start small with a TRL trainer on a small model, monitor KL divergence, reward, and sample outputs during training, and run held-out and regression evals — plus manual sample reading, since numbers hide weird behavior — before you ship.

What are the common failure modes of RL post-training?

All three methods share traps. Reward hacking: the model exploits flaws in the reward signal — use KL anchors, check samples by hand, and prefer verifiable rewards where you can; a learned reward model can be talked into high scores without good outputs. Length bias: preference data often favors longer answers and models learn to ramble — control for length in data and evaluation. Mode collapse: outputs lose diversity as training narrows the policy. Format gaming: the model satisfies a format check without solving the task — test with held-out problems. Capability regression: alignment training can hurt other skills — run broad evals before and after, including regression suites covering general skills and safety behavior. Weak SFT foundations: RL amplifies what the base already almost does. Team-level mistakes: skipping SFT, using DPO for everything when reasoning tasks usually need online exploration, trusting a reward model blindly, treating GRPO as one thing when its variants differ meaningfully, and evaluating only on the training distribution — hold out real test cases, and judge results with held-out preference or verifier-based tests and pairwise comparisons between old and new models using human reviewers or a calibrated judge, not training curves.

Wrapping This Up

RLHF, DPO, and GRPO solve related problems in different ways. RLHF offers flexibility at a high cost, DPO offers simplicity with offline limits, and GRPO offers on-policy learning where answers are verifiable.

Start with a strong SFT model, pick the simplest method that fits your feedback signal, and judge results with held-out evals instead of training curves. That discipline matters more than any acronym.

Related Articles

• Bias in LLM-as-a-Judge: Position, Verbosity, and Self-Preference
• Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement
• DeepEval Tutorial: Unit Test Your LLM Outputs in Python
• LLM Evaluation: Testing and Measuring Outputs the Right Way