GRPO Explained: Group Relative Policy Optimization for Reasoning Models

Contents

Grade on a curve, drop the critic

PPO needs a critic. The critic needs memory, training, and care. And then someone asked an irritating question: what if the model's own sibling answers could serve as the baseline instead?

That idea became Group Relative Policy Optimization, or GRPO. DeepSeek introduced it in DeepSeekMath, and DeepSeek-R1 pushed it into the spotlight. Today it's one of the most widely used algorithms for training reasoning models. This guide explains how GRPO works, what its known flaws are, how its variants fix them, and how to run it — the critic-free counterpart to our PPO walkthrough. It combines core-paper knowledge with current library docs and recent write-ups, and this area changes quickly, so check the papers and your library's docs before you build.

The Problem GRPO Solves

PPO-style RL needs an advantage estimate for each token: was this choice better or worse than expected? PPO gets that estimate from a value model (critic) that predicts expected reward. That critic is another large network to train and keep in memory, and it learns slowly, which makes advantage estimates noisy.

GRPO removes the critic. It gets its baseline from a simpler source: a group of answers to the same question.

The Core Idea

For each prompt, GRPO samples a group of responses, say 8 or 16. It scores every response, then compares each one to the group:

• A response that beats the group average gets a positive advantage.
• A response that falls below the average gets a negative advantage.
• The group itself serves as the baseline. No critic needed.

Think of a classroom where the teacher grades on a curve for each question. You don't need an external standard. You only need to know who did better than the others on that question.

def grpo_advantages(rewards, scale_std=True):
    """rewards: tensor of shape (G,) for one prompt's G responses."""
    centered = rewards - rewards.mean()
    return centered / (rewards.std() + 1e-6) if scale_std else centered

The original GRPO divides by the group's standard deviation. Later work questions that choice, as we'll see.

The GRPO Loop, Step by Step

TRL's documentation breaks GRPO into four main steps:

1. Generate completions. Sample G responses per prompt from the current policy.
2. Compute rewards and advantages. Score each response, then compute group-relative advantages.
3. Estimate KL divergence against a reference model, to keep the policy from drifting too far.
4. Compute the loss and update. Use a PPO-style clipped objective, applying each response's advantage to all of its tokens.

Notice what stays from PPO: the probability ratio and the clipping. What disappears: the value model and GAE. Every token in a response shares the same advantage, since the reward arrives once, at the end.

import torch

def grpo_loss(logp_new, logp_old, adv, mask, eps=0.2, norm="token"):
    """logp_*: (G, T) token log-probs. adv: (G,). mask: (G, T), 1 for real tokens."""
    ratio = torch.exp(logp_new - logp_old)
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps)
    per_token = -torch.min(ratio * adv[:, None], clipped * adv[:, None]) * mask
    if norm == "token":   # divide by total token count (DAPO-style)
        return per_token.sum() / mask.sum()
    # original GRPO: average within each response, then across the group
    return (per_token.sum(1) / mask.sum(1)).mean()

This sketch omits the KL term for clarity. Real implementations add it, though some recent setups reduce or drop it.

Rewards: The RLVR Connection

GRPO shines with Reinforcement Learning with Verifiable Rewards (RLVR). Instead of a learned reward model, simple programs check the output:

def format_reward(completion):
    tags = ["<think>", "</think>", "<answer>", "</answer>"]
    return 1.0 if all(t in completion for t in tags) else 0.0

def accuracy_reward(completion, gold):
    if "<answer>" not in completion or "</answer>" not in completion:
        return 0.0
    answer = completion.split("<answer>", 1)[1].split("</answer>", 1)[0]
    return 1.0 if answer.strip() == gold else 0.0

These checks work well for math (does the final answer match?) and code (do the tests pass?). Verifiers resist the kind of reward hacking that plagues learned reward models, though clever policies still find loopholes in sloppy checks.

How DeepSeek Used It

DeepSeek-R1-Zero applied reinforcement learning directly to a base model, with no supervised fine-tuning first, and reasoning behavior emerged. The production DeepSeek-R1 followed a more structured, multi-stage recipe that added cold-start data and supervised stages around the RL phases to improve readability and stability. The lesson for practitioners: pure RL can work, and a well-designed pipeline around it usually works better.

Figure 1: Each prompt is its own exam, and the group sets the curve

GRPO's Known Flaws

Research since the original papers found systematic biases in GRPO's formulation.

Length Bias

The original loss averages each response by its length. That inflates the gradient on short responses and dilutes it on long ones, so long wrong answers get under-penalized. A response-level normalization turns out to push models toward longer wrong answers.

Difficulty Bias

Dividing by the group's standard deviation over-weights questions where nearly all responses agree. Easy and very hard questions get amplified relative to medium ones, which skews learning.

Zero-Signal Groups

If every response in a group earns the same reward, all advantages are zero. The prompt teaches nothing, and you still paid for the generations. On hard problems with a low success rate, small groups often land entirely on failure.

Finite-Sample Effects

With a limited group size, group-relative estimates favor medium-difficulty prompts and under-learn from hard ones.

The GRPO Family: Fixes and Variants

• Dr. GRPO ("GRPO Done Right"): removes the per-response length normalization and the standard deviation scaling. The rest of the algorithm stays unchanged, and the result avoids both biases.
• DAPO (from ByteDance): a bundle of fixes found while scaling RL to long chain-of-thought. It adds clip-higher (a wider upper clip bound to encourage exploration), dynamic sampling (skip groups with no signal), token-level loss (sum over all tokens and divide by the total token count), and overlong reward shaping (penalize responses that run past a length limit).
• RLOO: a close cousin that uses a leave-one-out baseline. TRL ships it next to GRPO.
• GSPO and GMPO: variants that change how importance ratios combine: GSPO works at the sequence level, and GMPO takes a geometric mean of token ratios.

TRL's documentation exposes several of these as loss types, including original GRPO, a token-level variant, Dr. GRPO, and DAPO. It also notes that dynamic sampling isn't supported there. Check the current docs for exact option names.

"Use GRPO" is an underspecified instruction in 2026. Pick your normalization, your clipping, and your handling of empty-signal groups on purpose.

Running GRPO With TRL

Here's a minimal sketch. Argument names change between versions, so confirm them in the current docs:

from trl import GRPOConfig, GRPOTrainer

def correctness_reward(completions, answer, **kwargs):
    return [1.0 if extract_final(c) == a else 0.0
            for c, a in zip(completions, answer)]

trainer = GRPOTrainer(
    model="your-small-instruct-model",
    reward_funcs=[correctness_reward],
    args=GRPOConfig(
        output_dir="grpo-out",
        num_generations=8,          # group size
        learning_rate=1e-6,
        max_completion_length=512,
    ),
    train_dataset=dataset,          # columns: prompt, answer
)
trainer.train()

extract_final stands in for your own function that pulls the final answer out of a completion. For bigger runs, teams often move to frameworks built for scale, such as verl or OpenRLHF.

Practical Settings

• Group size: common values run from 8 to 16. Larger groups give better baselines and cost more generation.
• Learning rate: small, around 1e-6 for many setups.
• Clip range: around 0.2 as a default, with asymmetric ranges in DAPO-style setups.
• KL coefficient: some pipelines keep it small or remove it, while others rely on it for stability.
• Max completion length: set it deliberately, since longer limits raise cost and invite rambling.
• Prompt difficulty: aim for questions your model solves sometimes, not always or never. Those carry the most learning signal.

What to Monitor

• Mean reward and accuracy on training prompts, plus accuracy on a held-out set.
• Fraction of zero-variance groups: if it climbs, your prompts are too easy or too hard.
• Response length: sudden growth signals length exploitation.
• Entropy: a steep fall means the policy is collapsing onto narrow outputs.
• KL to the reference, if you use one.
• Format-compliance rate, separate from correctness.
• Samples: read completions every few hundred steps.

Failure Modes

• Reward hacking: the policy games your checker, for example by emitting the right format with a guessed answer. Test verifiers adversarially.
• Length growth: responses balloon without improving accuracy. Use token-level or constant normalization, and set length penalties.
• Entropy collapse: exploration dies. Consider clip-higher or other exploration aids.
• Wasted compute on empty groups: filter prompts by difficulty before training.
• Weak starting model: RL amplifies skills the model already shows occasionally. It rarely invents them.
• Overfitting to the training set: always evaluate on held-out problems with held-out evals.

GRPO vs PPO vs DPO

• PPO: learned critic, flexible, heavy.
• GRPO: group baseline, no critic, strong on verifiable tasks, costs many generations per prompt.
• DPO: offline preference pairs, cheap, limited by its dataset — see the DPO guide.

Use GRPO when answers are checkable and you want the model to explore. Use DPO when you have good preference pairs for style or safety. A 2026 paper even argues that GRPO connects closely to DPO and that very small groups can work surprisingly well. That's one result, so treat it as a hint for experiments, not a recipe.

Common Mistakes

• Starting from a base model that never gets the answer right — no signal, no learning.
• Using a leaky verifier — the policy finds the loophole.
• Ignoring normalization choices — length and difficulty biases creep in.
• Training only on easy prompts — groups go uniform, and the learning signal vanishes.
• Watching training reward only — hold out problems.
• Skipping the format check — models drift into unreadable reasoning without it.
• Throwing away zero-variance prompts — fix difficulty filtering instead of paying for dead groups.
• Judging by reward curves alone — pair them with held-out accuracy and manual reading.

Your Quick-Start Checklist

1. Pick a task with a reliable verifier (math answers or unit tests).
2. Start from a model that solves some problems in your set.
3. Filter prompts to a medium difficulty range.
4. Choose your loss normalization deliberately, with token-level or Dr. GRPO-style options in mind.
5. Set group size 8 to 16 and a small learning rate.
6. Monitor zero-variance groups, length, entropy, and held-out accuracy.
7. Test the verifier for loopholes before a long run.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is GRPO and how does it remove the critic?

Group Relative Policy Optimization (GRPO) replaces PPO's value model with a baseline computed from the model's own sibling answers. For each prompt, GRPO samples a group of responses (say 8 or 16), scores every one, then compares each to the group: a response that beats the group average gets a positive advantage, one that falls below gets a negative advantage, and the group itself serves as the baseline — no critic needed. Think of a classroom where the teacher grades on a curve for each question: you don't need an external standard, only who did better than the others on that question. The training loop has four steps: generate G completions per prompt from the current policy; compute rewards and group-relative advantages (reward minus group mean, divided by the group's standard deviation in the original formulation); estimate KL divergence against a reference model to keep the policy from drifting; then compute a PPO-style clipped loss and update, applying each response's advantage to all of its tokens. What stays from PPO: the probability ratio and the clipping (typically a 0.2 band). What disappears: the value model and GAE. Every token in a response shares the same advantage, since the reward arrives once, at the end. Removing the critic cuts memory and complexity, which is why GRPO became the default for training reasoning models after DeepSeekMath introduced it and DeepSeek-R1 pushed it into the spotlight.

What are verifiable rewards and how did DeepSeek use them?

GRPO shines with Reinforcement Learning with Verifiable Rewards (RLVR): instead of a learned reward model, simple programs check the output. A format check verifies that a completion contains the required thinking and answer tags, and an accuracy check extracts the answer between the answer tags and compares it to the gold solution. These checks work well for math (does the final answer match?) and code (do the unit tests pass?). Verifiers resist the kind of reward hacking that plagues learned reward models, though clever policies still find loopholes in sloppy checks — so test your verifier adversarially before a long run. DeepSeek's results show both extremes: DeepSeek-R1-Zero applied reinforcement learning directly to a base model with no supervised fine-tuning first, and reasoning behavior emerged; the production DeepSeek-R1 followed a more structured, multi-stage recipe that added cold-start data and supervised stages around the RL phases to improve readability and stability. The lesson for practitioners: pure RL can work, and a well-designed pipeline around it usually works better.

What are GRPO's known biases, and how do Dr. GRPO and DAPO fix them?

Research since the original papers found systematic biases in GRPO's formulation. Length bias: the original loss averages each response by its length, which inflates the gradient on short responses and dilutes it on long ones, so long wrong answers get under-penalized — a response-level normalization pushes models toward longer wrong answers. Difficulty bias: dividing by the group's standard deviation over-weights questions where nearly all responses agree, so easy and very hard questions get amplified relative to medium ones. Zero-signal groups: if every response in a group earns the same reward, all advantages are zero, the prompt teaches nothing, and you still paid for the generations — on hard problems with a low success rate, small groups often land entirely on failure. Finite-sample effects: with a limited group size, group-relative estimates favor medium-difficulty prompts and under-learn from hard ones. The fixes: Dr. GRPO (GRPO Done Right) removes the per-response length normalization and the standard deviation scaling, leaving the rest of the algorithm unchanged, which avoids both biases. DAPO, from ByteDance, bundles fixes found while scaling RL to long chain-of-thought — clip-higher (a wider upper clip bound to encourage exploration), dynamic sampling (skip groups with no signal), token-level loss (sum over all tokens and divide by the total token count), and overlong reward shaping (penalize responses that run past a length limit). RLOO is a close cousin using a leave-one-out baseline that TRL ships next to GRPO, and GSPO and GMPO change how importance ratios combine — sequence level for GSPO, a geometric mean of token ratios for GMPO. TRL's docs expose several as loss types, including original GRPO, a token-level variant, Dr. GRPO, and DAPO, and note that dynamic sampling isn't supported there, so check the current docs for exact option names.

How do you run GRPO with TRL, and how does it compare with PPO and DPO?

A minimal TRL sketch defines a reward function that returns one score per completion, then constructs a GRPOTrainer with a group size (num_generations, commonly 8 to 16), a small learning rate around 1e-6, and an explicit max_completion_length; the dataset needs prompt and answer columns, and any helper that extracts the final answer from a completion is yours to write. Argument names change between versions, so confirm them in the current docs, and for bigger runs teams often move to verl or OpenRLHF. Practical settings: group size 8 to 16 (larger groups give better baselines and cost more generation), learning rate around 1e-6, clip range around 0.2 as a default with asymmetric ranges in DAPO-style setups, a small or zero KL coefficient depending on your stability needs, a deliberate max completion length since longer limits raise cost and invite rambling, and prompt difficulty aimed at questions your model solves sometimes rather than always or never. Monitor mean reward and accuracy on training prompts plus a held-out set, the fraction of zero-variance groups (if it climbs, your prompts are too easy or too hard), response length, entropy, KL if used, format-compliance rate separately from correctness, and samples read every few hundred steps. Compared with the alternatives: PPO uses a learned critic, is flexible, and is heavy; GRPO uses a group baseline with no critic, is strong on verifiable tasks, and costs many generations per prompt; DPO trains offline on preference pairs, is cheap, and is limited by its dataset. Use GRPO when answers are checkable and you want the model to explore; use DPO when you have good preference pairs for style or safety. One 2026 paper even argues that GRPO connects closely to DPO and that very small groups can work surprisingly well — one result, so treat it as a hint for experiments, not a recipe.

Wrapping This Up

GRPO replaces PPO's critic with a simple idea: compare each answer to its siblings. That change cuts memory and complexity, and it pairs naturally with verifiable rewards for math and code.

The original recipe carries biases, and variants like Dr. GRPO and DAPO exist to fix them. Start small, filter prompts by difficulty, test your verifier, and evaluate on held-out problems. Do that, and GRPO gives you a practical path to reasoning gains without a reward model or a critic.

Related Articles

• PPO for RLHF: How Proximal Policy Optimization Fine-Tunes Language Models
• Reinforcement Learning for LLMs: Complete Guide to RLHF, DPO, and GRPO
• DPO Explained: Preference Tuning Without Reinforcement Learning
• Reward Models Explained: Teaching a Model What "Good" Looks Like