Process vs Outcome Reward Models: Rewarding Reasoning Steps

Contents

One scores the answer, the other scores the thinking

Your model writes a nine-step proof. Step 4 is nonsense, steps 5 through 9 politely build on the nonsense, and the final number is right anyway. Does that deserve a gold star? An outcome reward model (ORM) says yes. A process reward model (PRM) says "check step 4 again."

That disagreement is the whole topic. It's the supervision-design question that follows naturally from our reward models guide, and the simulations below are synthetic — no language model involved — so read them as mechanics.

What Each One Actually Scores

An outcome reward model looks at the finished solution and predicts one thing: is the final answer right? You train it on whole solutions labeled correct or incorrect. Labels are cheap, because for math or code you can often check the answer automatically.

A process reward model scores every step. You train it on step-level labels such as "this step is fine" or "this step goes wrong here." Labels are far more expensive, because someone (or something) has to judge the reasoning, not just the result.

Here is the practical difference:

• ORM: one score per solution, easy labels, blind to how the answer was reached.
• PRM: one score per step, costly labels, able to point at the exact step that broke.
• Shared use: both can rerank several sampled solutions (best-of-n) or serve as a reward signal for reinforcement learning.

What the Research Says

OpenAI's "Let's Verify Step by Step" compared the two directly on the MATH dataset. The authors report that process supervision significantly outperforms outcome supervision, and that their process-supervised model solves 78% of a representative subset of the MATH test set. They also found that active learning significantly improves the efficacy of process supervision, and they released PRM800K, a dataset of 800,000 step-level human labels.

Earlier work from Uesato et al. on GSM8K gives the more nuanced picture. Final-answer error improved from 16.8% to 12.7%. The reasoning-error rate among solutions with a correct final answer dropped from 14.0% to 3.4%. But the abstract also says pure outcome-based supervision produces similar final-answer error rates with less label supervision. Getting the reasoning right needs process-based supervision, or learned reward models that emulate it.

So the honest summary is not "PRMs always win." On final-answer accuracy the two can be close. On reasoning quality, process supervision has a clear edge.

Why Right Answers Hide Wrong Reasoning

How often does a wrong chain land on the right answer? A toy simulation shows how big the effect can get. Each step is correct with probability p, and a flawed chain still reaches the right answer with probability g. For an 8-step chain:

Step reliability p	Guess chance g	Chains that are fully valid	Correct answers with flawed reasoning
0.6	0.05	1.7%	74.5%
0.8	0.05	16.8%	19.9%
0.8	0.25	16.8%	55.4%
0.9	0.25	43.0%	24.9%

def flawed_but_correct(p, g, steps=8):
    """Share of correct final answers that came from a chain with an error."""
    valid = p ** steps
    correct = valid + (1 - valid) * g
    return round(valid, 3), round((correct - valid) / correct, 3)

print(flawed_but_correct(0.6, 0.05))   # (0.017, 0.745)
print(flawed_but_correct(0.8, 0.25))   # (0.168, 0.554)

The pattern is simple. When guessing is easy (multiple choice, small integer answers), an outcome check lets many flawed chains through. When guessing is hard, a right answer is decent evidence of sound reasoning. An ORM is least trustworthy exactly where lucky answers are common.

Best-of-n: Where Each Verifier Wins

Next, both verifiers got the same job: pick the best of n sampled solutions. Each verifier is right 90% of the time about the thing it judges. The ORM judges the final answer. The PRM judges each step and picks the solution with the most steps judged OK.

n	Random pick: answer right / reasoning valid	ORM pick	PRM pick
1	0.449 / 0.259	0.449 / 0.259	0.449 / 0.259
4	0.443 / 0.255	0.780 / 0.407	0.659 / 0.542
16	0.440 / 0.264	0.861 / 0.450	0.834 / 0.780
64	0.445 / 0.264	0.852 / 0.441	0.919 / 0.899

The conclusions, in order of how much they can be trusted:

1. On reasoning validity, the PRM wins at every n above 1. The ORM's pick plateaus around 45% valid reasoning, because it can't tell lucky chains from sound ones.
2. On final-answer accuracy, the ORM wins at small n (4 and 16). It judges the thing being measured directly.
3. At n = 64 the PRM wins on both. The ORM's accuracy stops improving, since it keeps selecting lucky chains.

There is one unfair detail worth naming. The PRM makes eight independent judgments per solution, and the ORM makes one. The PRM gets more information, so this table shows what step-level judging can buy, not a fair fight at equal cost. Real PRMs also make correlated mistakes, which the toy ignores.

Figure 1: Step-level labels catch the broken link; outcome labels only see the chain's end

Where Do Step Labels Come From?

Humans labeling every step is the expensive part. Math-Shepherd attacks it by building process supervision automatically, with no heavy manual annotation. The idea: to judge a step, sample several continuations from it and see how many reach the right answer.

The estimator models a prefix with no errors yet as worth p^(L−j) + g(1 − p^(L−j)), where j is the step number, and a prefix that already contains an error as worth about g. The question is how many rollouts k you need to tell them apart (measured as AUC, where 0.5 means a coin flip and 1.0 means perfect):

Step j	Good-prefix value	k=1	k=4	k=16	k=64
1	0.289	0.594	0.738	0.915	0.997
3	0.395	0.647	0.831	0.978	1.000
5	0.561	0.730	0.927	0.999	1.000
7	0.820	0.860	0.991	1.000	1.000

Two lessons follow. Early steps are the hardest to label, because a good start still has many chances to fail later. And more rollouts fix it, but you pay for every one:

def labeling_cost(steps, rollouts_per_step):
    """Extra sampled solutions needed to label one solution's steps."""
    return steps * rollouts_per_step

print(labeling_cost(steps=8, rollouts_per_step=16))   # 128

Labeling a single step at k=16 costs 16 extra solutions, and a full solution multiplies that by its step count.

In the paper's results, Mistral-7B with step-by-step PPO on Math-Shepherd rose from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH. Adding verification pushed it to 89.1% and 43.5%. Those are the authors' reported numbers, read from the abstract rather than reproduced here.

So Which One Should You Use?

Here is the honest take:

• Pick an ORM when answers are hard to guess, outcome checks are automatic, and you only need final-answer accuracy. It's cheap and often good enough.
• Pick a PRM when answers are easy to guess, reasoning quality matters to your users, or you plan to optimize hard against the verifier. That last case is where the simulation showed the ORM stalling — the same pressure dynamics covered in the reward hacking guide.
• Consider automated step labels if human annotation is out of budget, and budget for the rollouts.
• Check your data first. Sample 50 "correct" solutions and read the reasoning.

def lucky_guess_share(correct_solutions, flagged_visible_flaws):
    """Among correct solutions, the share whose reasoning shows a visible flaw."""
    return round(flagged_visible_flaws / correct_solutions, 2)

print(lucky_guess_share(50, 13))   # 0.26 — a quarter of them are lucky guesses

If a quarter of your correct solutions are lucky guesses, you've found your answer.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between an outcome reward model and a process reward model?

An outcome reward model (ORM) looks at the finished solution and predicts one thing: is the final answer right? You train it on whole solutions labeled correct or incorrect, and labels are cheap because for math or code you can often check the answer automatically. A process reward model (PRM) scores every step, trained on step-level labels such as this step is fine or this step goes wrong here; labels are far more expensive because someone or something has to judge the reasoning, not just the result. In practice that means: ORM gives one score per solution, easy labels, and is blind to how the answer was reached; PRM gives one score per step, costly labels, and can point at the exact step that broke. Both can be used the same way — to rerank several sampled solutions (best-of-n) or as a reward signal for reinforcement learning. The disagreement between them shows up most clearly on solutions that are right for the wrong reason: an ORM rewards a nine-step proof whose step four is nonsense, while a PRM flags step four.

What does the research say about process versus outcome supervision?

OpenAI's Let's Verify Step by Step compared the two directly on the MATH dataset: the authors report that process supervision significantly outperforms outcome supervision, and that their process-supervised model solves 78% of a representative subset of the MATH test set. They also found that active learning significantly improves the efficacy of process supervision, and released PRM800K, a dataset of 800,000 step-level human labels. Earlier work from Uesato et al. on GSM8K gives the more nuanced picture — final-answer error improved from 16.8% to 12.7%, and the reasoning-error rate among solutions with a correct final answer dropped from 14.0% to 3.4%, but their abstract also says pure outcome-based supervision produces similar final-answer error rates with less label supervision. So the honest summary is not that PRMs always win: on final-answer accuracy the two can be close, while on reasoning quality process supervision has a clear edge, and getting the reasoning right needs process-based supervision or learned reward models that emulate it.

How often do correct final answers come from flawed reasoning?

A toy simulation of the effect — where each step is correct with probability p and a flawed chain still reaches the right answer with probability g — shows how big the gap can get over an 8-step chain. At p = 0.6 with a 5% guess chance, only 1.7% of chains are fully valid while 74.5% of correct answers come from flawed reasoning. At p = 0.8 with a 5% guess chance, those numbers are 16.8% and 19.9%; raise the guess chance to 25% at the same step reliability and 55.4% of correct answers come from flawed chains. At p = 0.9 with a 25% guess chance, 43.0% of chains are fully valid and 24.9% of correct answers are flawed. The pattern is simple: when guessing is easy (multiple choice, small integer answers), an outcome check lets many flawed chains through, and when guessing is hard a right answer is decent evidence of sound reasoning. An ORM is least trustworthy exactly where lucky answers are common, which is why reading your own correct outputs is the fastest diagnostic — sample 50 correct solutions and read the reasoning; if a quarter of them are lucky guesses, you have your answer.

Where do process labels come from, and how do PRMs and ORMs compare under selection pressure?

Humans labeling every step is the expensive part. Math-Shepherd attacks it by building process supervision automatically with no heavy manual annotation: to judge a step, sample several continuations from it and see how many reach the right answer. A prefix with no errors yet has value p^(L−j) + g(1 − p^(L−j)), where j is the step number, while a prefix that already contains an error is worth about g. The question is how many rollouts k you need to tell them apart, measured as AUC where 0.5 is a coin flip and 1.0 is perfect: early steps are the hardest to label, because a good start still has many chances to fail later, and more rollouts fix it but you pay for every one — labeling a single step at k = 16 costs 16 extra solutions, and a full solution multiplies that by its step count. In the paper's results, Mistral-7B with step-by-step PPO on Math-Shepherd rose from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH, and adding verification pushed it to 89.1% and 43.5% (the authors' reported numbers, read from the abstract rather than reproduced). Under selection pressure, a simulation where each verifier is right 90% of the time about the thing it judges found: the PRM wins on reasoning validity at every n above 1, while the ORM's pick plateaus around 45% valid reasoning because it cannot tell lucky chains from sound ones; the ORM wins on final-answer accuracy at small n (4 and 16) because it judges the measured thing directly; and at n = 64 the PRM wins on both, since the ORM's accuracy stops improving as it keeps selecting lucky chains. One caveat matters — the PRM makes eight independent judgments per solution while the ORM makes one, so that table shows what step-level judging can buy, not a fair fight at equal cost, and real PRMs also make correlated mistakes.

Wrapping This Up

An outcome reward model asks whether the answer is right. A process reward model asks whether the thinking is right. Published results suggest process supervision wins on reasoning quality, while outcome supervision stays competitive on final answers for less labeling effort. The simulation agrees on the shape: the ORM leads at low selection pressure, and the PRM pulls ahead as pressure grows.

Before you pick a side, read some of your own "correct" outputs. Then decide which kind of correct you're paying for.

Related Articles

• Reward Models Explained: Teaching a Model What "Good" Looks Like
• Reward Hacking in LLMs: Why Models Game Their Rewards
• RLVR: Reinforcement Learning with Verifiable Rewards Explained
• How Reasoning Models Are Trained: RL and Chain-of-Thought