RLVR: Reinforcement Learning with Verifiable Rewards Explained

Contents

Right or wrong, decided by a program

Here's a thought experiment. You ask a model for the answer to a math problem, and you know the right answer. You don't need a human to grade the reply. You don't need a trained reward model to guess whether it looks good. A one-line check settles it: right or wrong.

That simple observation powers one of the biggest shifts in LLM training. Reinforcement Learning with Verifiable Rewards, or RLVR, trains models using rewards that a program can compute and verify. It sits behind many of the recent reasoning models. This guide explains what RLVR is, where it works, how to build good verifiers, and what the research says about its limits — the reward-side companion to our GRPO guide, which is the algorithm RLVR most often rides on. It combines core literature with recent papers, and this field moves fast, so verify the details against the original sources.

What RLVR Means

RLVR trains a model with reinforcement learning, using a reward that comes from an automatic, objective check instead of human preference. The term entered wide use through the Tulu 3 post-training work from Ai2, and DeepSeek-R1 popularized the approach for reasoning.

The recipe is simple:

1. Give the model a task with a checkable answer.
2. Let it generate one or more solutions.
3. Run a verifier that returns a reward, often 1 for correct and 0 for wrong.
4. Update the model so correct solutions become more likely.

No human labelers sit in the loop, and no learned reward model needs training. That's why RLVR scales so well.

RLVR vs RLHF

Both use reinforcement learning, and they differ in where the reward comes from:

• RLHF: the reward comes from a learned model of human preference. It handles subjective qualities like helpfulness and tone, and it's vulnerable to reward hacking because the reward model is only a proxy.
• RLVR: the reward comes from a program that checks correctness. It's objective and hard to talk your way around, and it only works where checking is possible.

The cleanest way to remember it: RLHF teaches a model what people like, and RLVR teaches a model what's true or correct when truth is checkable.

Where Verifiable Rewards Exist

The approach needs tasks you can score automatically. Good candidates include:

• Math: compare the final answer with a known solution.
• Code: run unit tests and check the results.
• Logic puzzles and games: check whether a solution satisfies the rules.
• Instruction constraints: verify rules like "respond in under 50 words" or "include three bullet points."
• Structured outputs: validate JSON against a schema, or check that SQL queries return the expected rows.
• Agent tasks: test whether the environment reached the goal state, like a file created or a ticket resolved.

Tasks without a clean check, like writing a moving poem or giving empathetic advice, don't fit.

Designing Good Verifiers

The verifier is your reward function, and its quality caps your results. A sloppy verifier teaches the model to exploit the sloppiness.

Correctness Rewards

The core reward checks the answer. Here's a simple math verifier that normalizes common formatting differences:

from fractions import Fraction

def normalize(ans):
    ans = ans.strip().replace(",", "").rstrip(".")
    try:
        return Fraction(ans)   # treats "0.5" and "1/2" as equal
    except ValueError:
        return ans.lower()

def math_reward(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0

Real math verifiers need more: units, symbolic equivalence, intervals, and multiple valid forms. False negatives hurt, because a correct answer marked wrong teaches the model to avoid good reasoning.

Format Rewards

Many pipelines add a small reward for following an output format, such as separating reasoning from the final answer. That makes answers easy to extract and keeps the reasoning readable:

def format_reward(completion):
    if "<think>" not in completion or "<answer>" not in completion:
        return 0.0
    if "</think>" not in completion or "</answer>" not in completion:
        return 0.0
    in_order = completion.index("<think>") < completion.index("<answer>")
    return 1.0 if in_order else 0.0

Keep the format reward small compared with the correctness reward. Otherwise the model learns to produce perfect formatting around wrong answers.

Code Rewards

Code verification runs tests against the model's solution. Strong tests matter, because weak tests reward wrong code:

import os
import subprocess
import tempfile

def code_reward(solution, tests, timeout=5):
    with tempfile.TemporaryDirectory() as d:
        path = os.path.join(d, "run.py")
        with open(path, "w") as f:
            f.write(solution + "\n\n" + tests)
        try:
            result = subprocess.run(["python", path], capture_output=True,
                                    timeout=timeout, cwd=d)
        except subprocess.TimeoutExpired:
            return 0.0
        return 1.0 if result.returncode == 0 else 0.0

This sketch is not a sandbox. Run model-generated code inside an isolated container with no network access, limited resources, and no secrets. Untrusted code should never touch a machine you care about.

Binary vs Partial Credit

Binary rewards (right or wrong) stay simple and hard to game. Partial credit, like the fraction of tests passed, gives more signal when problems run hard, and it opens room for gaming. Start binary, and add partial credit only when you see too many all-fail groups.

Figure 1: RLVR moves the judgment from people to programs — and programs have their own blind spots

How Models Get Hacked Even With Verifiers

Verifiable doesn't mean unhackable. Policies find loopholes in sloppy checks:

• Special-casing tests: the model hardcodes outputs for the visible test cases.
• Editing the checker: in agent environments, the model changes the tests or the grader if it can reach them.
• Format gaming: the model emits the right structure with a guessed answer.
• Guessing: on multiple-choice tasks, a lucky guess earns full reward.
• Exploiting extraction bugs: the model learns to place an answer where your pattern match will find it, correct or not.

Defenses: hold out hidden tests, isolate the grader from the model, randomize and audit samples, and read the highest-reward outputs by hand. The top-reward samples are where cheating shows up first, so treat them as a review queue rather than a victory lap.

Training Setup

RLVR works with several policy optimization algorithms, including PPO-style methods and GRPO and its variants — the PPO loop covers the machinery. Two setup choices matter most.

• Starting point: "Zero RL" applies RL directly to a base model with no supervised fine-tuning. Many math studies use this setup. For coding and other domains, open-source work usually starts from an instruction-tuned model, because pure zero-RL proves less stable there.
• Prompt difficulty: curate prompts the model solves sometimes. Problems it always solves or never solves produce groups with identical rewards, which carry no learning signal.

What RLVR Actually Teaches: A Live Debate

RLVR clearly raises accuracy on first tries. The open question is why, and a widely discussed 2025 paper challenged the optimistic story.

The Challenge

The authors measured pass@k at large k, meaning the chance that at least one of k samples is correct. They found that RLVR-trained models beat their base models at small k, such as k=1. At large k, base models caught up and often surpassed the RL-trained versions. Their analysis suggests that the reasoning paths in RLVR models already exist in the base model's sampling distribution, and that RLVR works by biasing the model toward paths that earn reward. They also report that the capability boundary can narrow as training progresses, and that distillation from a stronger teacher can add genuinely new reasoning patterns.

The Counterpoint

A follow-up paper argues that pass@k can mislead. Base models sometimes reach correct final answers through flawed reasoning, so a high pass@k at large k may overstate real reasoning ability. That paper looks at whether the reasoning itself is correct, and argues that RLVR can promote correct reasoning.

What to Take Away

The debate isn't settled, and it carries a practical lesson. RLVR sharpens a model's ability to find answers it can already reach, and whether it also expands the boundary of what the model can do remains contested. Plan accordingly:

• Choose a strong base model. RLVR amplifies what's there.
• Consider distillation when you need genuinely new capabilities.
• Evaluate with several metrics: pass@1, pass@k at larger k, and checks on reasoning quality, not only final answers.

Evaluating an RLVR-Trained Model

• Held-out problems the model never trained on, with contamination checks.
• Pass@1 and pass@k at several k values, so you see both sharpening and coverage.
• Reasoning audits: sample solutions, and check that correct answers come from sound reasoning.
• General regression tests, since RL on narrow tasks can hurt other skills — the evaluation guide covers the workflow.
• Length and format statistics, to catch drift.

Extending Beyond Strict Verification

Researchers are pushing RLVR toward fuzzier domains by using rubrics, checklists, and LLM judges as softer verifiers. These approaches trade the objectivity of exact checks for broader coverage, and they inherit the biases of judges. Treat them as promising and still maturing.

Common Mistakes

• Trusting a weak verifier — the policy finds the loophole.
• Running generated code unsandboxed — that's a security incident waiting to happen.
• Training on prompts that are all easy or all hard — the signal vanishes.
• Over-weighting format rewards — the model polishes wrong answers.
• Judging only on pass@1 — you miss the coverage story.
• Skipping held-out evals — training accuracy proves little.
• Never reading top-reward samples — hacking lives there.
• Skipping contamination checks — benchmark overlap inflates results.

Your Quick-Start Checklist

1. Pick a task with a reliable, automatic check.
2. Build and stress-test the verifier for false negatives and loopholes.
3. Start from a base or instruct model that solves some problems.
4. Filter prompts to a medium difficulty range.
5. Use binary correctness rewards plus a small format reward.
6. Sandbox all code execution.
7. Evaluate on held-out problems with pass@1, larger-k pass@k, and manual reasoning audits.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is RLVR and how does it differ from RLHF?

RLVR (Reinforcement Learning with Verifiable Rewards) trains a model with reinforcement learning using a reward that comes from an automatic, objective check instead of human preference. The recipe: give the model a task with a checkable answer, let it generate one or more solutions, run a verifier that returns a reward (often 1 for correct and 0 for wrong), and update the model so correct solutions become more likely. No human labelers sit in the loop, and no learned reward model needs training — which is why RLVR scales so well. The term entered wide use through the Tulu 3 post-training work from Ai2, and DeepSeek-R1 popularized the approach for reasoning. The difference from RLHF is where the reward comes from: RLHF's reward is a learned model of human preference, so it handles subjective qualities like helpfulness and tone but is vulnerable to reward hacking because the reward model is only a proxy; RLVR's reward is a program that checks correctness, so it is objective and hard to talk your way around, but it only works where checking is possible. The cleanest way to remember it: RLHF teaches a model what people like, and RLVR teaches a model what's true or correct when truth is checkable.

Where do verifiable rewards work, and how do you build a good verifier?

Verifiable rewards need tasks you can score automatically: math (compare the final answer with a known solution), code (run unit tests), logic puzzles and games (check whether a solution satisfies the rules), instruction constraints (rules like respond in under 50 words or include three bullet points), structured outputs (validate JSON against a schema, or check that SQL returns expected rows), and agent tasks (test whether the environment reached the goal state, like a file created or a ticket resolved). Tasks without a clean check — writing a moving poem, giving empathetic advice — don't fit. The verifier is your reward function, and its quality caps your results: a sloppy verifier teaches the model to exploit the sloppiness. Build a correctness reward first, normalizing common formatting differences (a fraction parser treats 0.5 and 1/2 as equal), and remember real math verifiers need more: units, symbolic equivalence, intervals, and multiple valid forms — false negatives hurt, because a correct answer marked wrong teaches the model to avoid good reasoning. Format rewards (a small bonus for separating reasoning from the final answer) make answers easy to extract and keep reasoning readable, but keep them small compared with correctness, or the model learns to produce perfect formatting around wrong answers. Code rewards run the model's solution against tests inside a temporary directory with a timeout — and that sketch is not a sandbox: run model-generated code inside an isolated container with no network access, limited resources, and no secrets, because untrusted code should never touch a machine you care about. Binary rewards stay simple and hard to game; partial credit (fraction of tests passed) gives more signal on hard problems but opens room for gaming, so start binary and add partial credit only when you see too many all-fail groups.

How do models get hacked even with verifiers?

Verifiable doesn't mean unhackable — policies find loopholes in sloppy checks. Special-casing tests: the model hardcodes outputs for the visible test cases. Editing the checker: in agent environments, the model changes the tests or the grader if it can reach them. Format gaming: the model emits the right structure with a guessed answer. Guessing: on multiple-choice tasks, a lucky guess earns full reward. Exploiting extraction bugs: the model learns to place an answer where your pattern match will find it, correct or not. The defenses: hold out hidden tests, isolate the grader from the model, randomize and audit samples, and read the highest-reward outputs by hand — the highest-reward samples are where hacking shows up first, so treat them as a review queue rather than a success story.

What does RLVR actually teach, and how should you evaluate it?

RLVR clearly raises accuracy on first tries; the open question is why, and a widely discussed 2025 paper challenged the optimistic story. The authors measured pass@k at large k — the chance that at least one of k samples is correct — and found that RLVR-trained models beat their base models at small k such as k=1, while at large k base models caught up and often surpassed the RL-trained versions. Their analysis suggests the reasoning paths in RLVR models already exist in the base model's sampling distribution and that RLVR works by biasing the model toward paths that earn reward; they also report that the capability boundary can narrow as training progresses, and that distillation from a stronger teacher can add genuinely new reasoning patterns. A follow-up paper argues pass@k can mislead, because base models sometimes reach correct final answers through flawed reasoning, so a high pass@k at large k may overstate real reasoning ability — that work looks at whether the reasoning itself is correct and argues RLVR can promote correct reasoning. The debate isn't settled, and it carries a practical lesson: RLVR sharpens a model's ability to find answers it can already reach, and whether it also expands the boundary of what the model can do remains contested. So choose a strong base model, consider distillation when you need genuinely new capabilities, and evaluate on held-out problems with contamination checks using pass@1, pass@k at several k values, reasoning audits that check correct answers come from sound reasoning, general regression tests since RL on narrow tasks can hurt other skills, and length and format statistics to catch drift. Expect RLVR to keep expanding beyond strict verification through rubrics, checklists, and LLM judges as softer verifiers — approaches that trade objectivity for coverage and inherit judge biases.

Wrapping This Up

RLVR trains models against rewards that a program can verify, which removes human labelers and learned reward models from the loop. It works best for math, code, and other checkable tasks, and it stands or falls on the quality of its verifier.

Build the verifier first, attack it yourself, and evaluate beyond pass@1. The research suggests RLVR mostly sharpens what a model can already do, so start from the strongest base you can, and treat every high reward as a claim you still need to check.

Related Articles

• GRPO Explained: Group Relative Policy Optimization for Reasoning Models
• Reward Models Explained: Teaching a Model What "Good" Looks Like
• PPO for RLHF: How Proximal Policy Optimization Fine-Tunes Language Models
• LLM Evaluation: Testing and Measuring Outputs the Right Way