Reward Models Explained: Teaching a Model What “Good” Looks Like

Contents

The learned judge at the heart of alignment training

Here's a puzzle. You want a language model to write better answers, but "better" has no formula. No equation tells you whether a reply is helpful, honest, or well written.

So you build a model that judges instead. It reads a prompt and a response, then outputs a score. That scorer is a reward model, and it sits at the heart of modern alignment training. This guide explains what reward models are, how they're built, how they fail, and how to keep them honest — the natural follow-up to our RLHF deep dive, where the reward model is stage two of the pipeline. The failure research here combines core literature with 2026 work on reward model robustness, so verify specifics against the original papers before you cite them.

What a Reward Model Does

A reward model takes a prompt and a response and returns a number. A higher number means "people would prefer this." You use that number as a training signal, a ranking tool, or a quality filter.

Think of it as a stand-in for human judgment. Humans can't read millions of model outputs, but a reward model can score them all, quickly and consistently — with caveats we'll get to.

Reward models solve a specific problem: tasks where no program can check the answer. For math or code, a verifier can test correctness directly. For helpfulness, tone, or safety, you need a learned judge.

How a Reward Model Is Built

The Architecture

Most reward models start from a pretrained language model. Builders remove the usual next-word prediction layer and add a small head that outputs a single number. The model reads the prompt and response together, then scores the pair:

from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("your-base-model")
rm = AutoModelForSequenceClassification.from_pretrained("your-base-model", num_labels=1)

def score(prompt, response):
    inputs = tok(prompt + response, return_tensors="pt")
    return rm(**inputs).logits[0, 0]

This sketch shows the idea. Real setups format chat templates carefully and handle batching and padding.

The Training Data

Reward models learn from preference data. Each example contains a prompt, a preferred (chosen) response, and a less preferred (rejected) response. Human labelers usually make the call, though AI feedback increasingly supplies labels too.

Data quality decides everything here. Noisy labels, vague guidelines, and narrow prompts all show up later as strange behavior — the annotation discipline applies with full force.

The Training Objective

The model learns that the chosen response should score higher than the rejected one. The standard loss comes from the Bradley-Terry model of pairwise preferences:

import torch.nn.functional as F

def reward_model_loss(r_chosen, r_rejected):
    return -F.logsigmoid(r_chosen - r_rejected).mean()

Notice what the loss sees: only the difference between two scores. The absolute scale of the output stays arbitrary. A reward model might output scores around 3 or around -40, and both can rank perfectly. So you must normalize scores before you combine them with other reward signals.

The Reward Model Family

The term covers several different designs.

Scalar Preference Reward Models

These are the classic version: one number per response, trained on preference pairs. They're fast and cheap to run, but they don't explain their scores.

Outcome Reward Models (ORMs)

An ORM scores the final result of a solution, often trained on whether the final answer was correct. Surveys describe ORMs as comparatively resistant to reward hacking when they rely on ground-truth verification.

Process Reward Models (PRMs)

A PRM scores each reasoning step. That gives denser feedback, since the model learns which step went wrong, not only that the answer failed. Researchers use PRMs for reward-guided decoding and for scaling test-time compute. The catch: surveys report that PRMs prove more susceptible to length and verbosity hacking than ORMs, because step-level signals are noisier.

Generative Reward Models

Instead of outputting a number directly, a generative reward model writes a critique or reasoning first, then gives a verdict. Some approaches, like RM-R1, generate a rubric or reference solution before they judge, and reports describe gains over larger scalar models on reward model benchmarks. The explanation also makes the judgment easier to inspect.

LLM Judges

An LLM judge swaps the trained scoring head for a prompt. You tell a capable model what to grade and how, and it returns a verdict. This approach needs no training data, which makes it a fast starting point. Our judge bias guide covers its weak spots — position, verbosity, and self-preference included.

Rule-Based Verifiers

When correctness is checkable, a program beats a model. Unit tests, answer matching, and format checks give clean, hard-to-fool rewards. Use them wherever you can, and save learned reward models for the rest — the same instinct that drives verifiable rewards in GRPO.

Figure 1: Under the hood a reward model is a familiar language model with the output layer swapped for one number

Where Reward Models Get Used

Reward models do more than power RLHF:

• Reinforcement learning: the reward model scores policy outputs during RLHF-style training.
• Best-of-N sampling: generate several answers, score them all, and return the highest.
• Rejection sampling for training data: keep only high-scoring generations for further fine-tuning.
• Data filtering: score and prune low-quality training examples.
• Evaluation: compare model versions using reward scores as a proxy.

Best-of-N makes a good first experiment, because it needs no RL training at all:

def best_of_n(prompt, generate, score, n=8):
    candidates = [generate(prompt) for _ in range(n)]
    return max(candidates, key=lambda c: score(prompt, c))

How to Evaluate a Reward Model

A reward model deserves its own evaluation before you trust it with a training run.

• Held-out pair accuracy: on comparisons the model never saw, how often does it pick the human-preferred response? This number has a ceiling, since human labelers often disagree with each other on a meaningful share of pairs.
• Reward model benchmarks: public suites test reward models across categories like chat, reasoning, and safety. Use them for rough comparisons, and remember that scores can diverge from real downstream results.
• Perturbation tests: change style, length, or formatting without changing meaning, and check whether scores stay stable. Add logical errors that read fluently, and check whether scores drop.
• Downstream tests: run best-of-N or a small training run, then evaluate the results with humans or an independent judge.

The perturbation tests matter most. A reward model that loves long answers will teach your policy to ramble.

How Reward Models Fail

Reward Overoptimization

The reward model approximates human preference, and a strong optimizer exploits the gaps. As training pushes the proxy score higher, true quality eventually peaks and then declines. Research on scaling laws for this effect found that larger reward models reach higher peak quality and overoptimize less, though the effect never fully disappears.

Length and Style Bias

Reward models often favor longer, more polished, more confident answers, since annotators did too. The policy learns to pad.

Sycophancy

When raters favor agreeable answers, the reward model learns to reward agreement, even when the user is wrong.

Fluency Over Correctness

A 2026 study of process reward models makes this failure vivid. The authors report that state-of-the-art PRMs react little to surface style changes yet inconsistently catch logically corrupted reasoning. When they trained policies against PRM feedback on competition math problems, the policies reached reward scores above 0.9 while true accuracy stayed below 4%, and the authors attribute 43% of the reward gains to stylistic shortcuts. They conclude that current PRMs behave more like fluency detectors than reasoning verifiers. That's one study, but it carries a clear warning.

Distribution Shift

A reward model learns from responses that earlier models produced. As the policy improves and drifts, it generates outputs the reward model never saw, and scores there turn unreliable.

How to Keep Reward Models Honest

No single fix works, so teams stack several:

• Use a KL penalty to limit how far the policy drifts from its starting point.
• Refresh the reward model with new comparisons drawn from the current policy's outputs.
• Train an ensemble of reward models whose errors differ. One agent-focused PRM study found that ensembles alone didn't stop hacking in their setup, so treat ensembles as one tool among several.
• Prefer verifiers wherever answers are checkable.
• Use larger reward models, which overoptimize less in the scaling studies.
• Try generative reward models that reason before they score.
• Stop early. Watch held-out quality and halt when it peaks, even if proxy reward keeps climbing.
• Audit by hand. Read samples at regular intervals.

Build Your Own: A Practical Outline

1. Define "good" with written guidelines that labelers can follow consistently.
2. Collect preference pairs from realistic prompts, with a mix of easy and hard cases.
3. Measure labeler agreement, and fix guideline gaps.
4. Train a scalar reward model with the pairwise loss, starting from a strong base model.
5. Evaluate on held-out pairs and perturbation tests.
6. Try best-of-N to see whether higher scores track better answers.
7. Only then use it for RL, with a KL penalty and held-out monitoring.

Common Mistakes

• Trusting the score as truth — it's a proxy for preference, not for correctness.
• Skipping perturbation tests — length and style bias hide until they bite.
• Using a learned reward where a verifier works — verifiers resist hacking better.
• Ignoring distribution shift — refresh the reward model as the policy changes.
• Mixing raw scores from different models — normalize first, since offsets are arbitrary.
• Watching only the reward curve — rising reward with falling quality means hacking.
• Skipping labeler-agreement checks — noisy preferences make a noisy judge.
• Never testing the judge itself — evaluate the reward model the same way you evaluate any model.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is a reward model and how is it trained?

A reward model takes a prompt and a response and returns a number where a higher number means "people would prefer this." You use that number as a training signal, a ranking tool, or a quality filter — a stand-in for human judgment, since humans can't read millions of model outputs but a reward model can score them all quickly and consistently. It solves a specific problem: tasks where no program can check the answer. For math or code, a verifier tests correctness directly; for helpfulness, tone, or safety, you need a learned judge. Most reward models start from a pretrained language model: builders remove the usual next-word prediction layer and add a small head that outputs a single number, reading the prompt and response together before scoring the pair. Training uses preference data — each example contains a prompt, a preferred (chosen) response, and a less preferred (rejected) response, usually judged by human labelers, with AI feedback increasingly supplying labels too. The standard loss comes from the Bradley-Terry model of pairwise preferences: -logsigmoid(r_chosen - r_rejected) averaged over the batch, which pushes the chosen response's score above the rejected one's. Notice what the loss sees: only the difference between two scores, so the absolute scale stays arbitrary — a reward model might output scores around 3 or around -40 and both can rank perfectly, so normalize scores before combining them with other reward signals.

What is the difference between ORM, PRM, and a generative reward model?

The term covers several designs. Scalar preference reward models are the classic version: one number per response, trained on preference pairs — fast and cheap to run, but they don't explain their scores. Outcome Reward Models (ORMs) score the final result of a solution, often trained on whether the final answer was correct; surveys describe ORMs as comparatively resistant to reward hacking when they rely on ground-truth verification. Process Reward Models (PRMs) score each reasoning step instead of only the outcome, giving denser feedback — the model learns which step went wrong, not only that the answer failed — and researchers use PRMs for reward-guided decoding and for scaling test-time compute; the catch is that surveys report PRMs prove more susceptible to length and verbosity hacking than ORMs, because step-level signals are noisier. Generative reward models don't output a number directly: they write a critique or reasoning first, then give a verdict — some approaches, like RM-R1, generate a rubric or reference solution before judging, and reports describe gains over larger scalar models on reward model benchmarks, with the explanation making the judgment easier to inspect. Two more family members: LLM judges, which swap the trained scoring head for a prompt to a capable model (no training data needed, a fast starting point, with known weak spots around bias), and rule-based verifiers, where unit tests, answer matching, and format checks give clean, hard-to-fool rewards — use them wherever correctness is checkable and save learned reward models for the rest.

How do reward models fail?

Several failure modes show up repeatedly. Reward overoptimization: the reward model approximates human preference, and a strong optimizer exploits the gaps — as training pushes the proxy score higher, true quality eventually peaks and then declines; research on scaling laws for this effect found that larger reward models reach higher peak quality and overoptimize less, though the effect never fully disappears. Length and style bias: reward models often favor longer, more polished, more confident answers, since annotators did too, and the policy learns to pad. Sycophancy: when raters favor agreeable answers, the reward model learns to reward agreement even when the user is wrong. Fluency over correctness: a 2026 study of process reward models makes this vivid — the authors report that state-of-the-art PRMs react little to surface style changes yet inconsistently catch logically corrupted reasoning, and when they trained policies against PRM feedback on competition math problems, the policies reached reward scores above 0.9 while true accuracy stayed below 4%, with the authors attributing 43% of the reward gains to stylistic shortcuts; they conclude that current PRMs behave more like fluency detectors than reasoning verifiers. That's one study, but it carries a clear warning. Distribution shift: a reward model learns from responses that earlier models produced — as the policy improves and drifts, it generates outputs the reward model never saw, and scores there turn unreliable.

How do you evaluate a reward model before trusting it?

A reward model deserves its own evaluation before you trust it with a training run. Held-out pair accuracy: on comparisons the model never saw, how often does it pick the human-preferred response — this number has a ceiling, since human labelers often disagree with each other on a meaningful share of pairs, so measure agreement too. Reward model benchmarks: public suites test reward models across categories like chat, reasoning, and safety — useful for rough comparisons, but scores can diverge from real downstream results. Perturbation tests: change style, length, or formatting without changing meaning and check whether scores stay stable; add logical errors that read fluently and check whether scores drop — practitioners consider these the most informative test, because a reward model that loves long answers will teach your policy to ramble. Downstream tests: run best-of-N (generate several answers, score them all, return the highest — a good first experiment since it needs no RL training at all) or a small training run, then evaluate the results with humans or an independent judge. When building your own: define "good" with written guidelines, collect preference pairs from realistic prompts with a mix of easy and hard cases, measure labeler agreement and fix guideline gaps, train the scalar model from a strong base, test on held-out pairs and perturbations, and only then use it for RL — with a KL penalty, refreshed comparisons drawn from the current policy's outputs, and hand-read samples at regular intervals.

Wrapping This Up

A reward model turns human judgment into a number a training loop can use. It's powerful, and it stays a proxy: it measures what raters tended to prefer, and optimizers will find every gap between that and what you actually want.

Use verifiers where you can, test reward models for length and style bias, and watch an independent quality measure while you train. A reward model works best when you treat it as a useful but fallible judge.

Related Articles

• RLHF Explained: How Human Feedback Trains ChatGPT-Style Models
• Reinforcement Learning for LLMs: Complete Guide to RLHF, DPO, and GRPO
• Bias in LLM-as-a-Judge: Position, Verbosity, and Self-Preference
• Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement