How Reasoning Models Are Trained: RL and Chain-of-Thought

Contents

Pause, think, verify, answer — trained in, not born with

Ask a regular chatbot a hard math question, and it starts answering immediately. Ask a reasoning model the same question, and it pauses to think: it writes out steps, checks them, backs up when something looks wrong, and only then answers.

That behavior doesn't come free with a bigger model. Someone trained it in, and the recipe combines two ingredients: chain-of-thought reasoning and reinforcement learning. This guide walks through how reasoning models get built, using DeepSeek-R1 as the best-documented reference, and explains what each training stage contributes — the pipeline view that ties together our RLVR and GRPO guides. It combines literature knowledge with secondary summaries of the R1 paper, so check the original papers before you cite specifics.

What Makes a Model a "Reasoning Model"

A reasoning model generates a long internal chain of thought before it commits to a final answer. Instead of jumping from question to answer in a few tokens, it spends hundreds or thousands of tokens working through the problem.

That shift opens a new scaling dimension: test-time compute. Traditional scaling improved models by training bigger networks on more data. Reasoning models also improve by thinking longer at inference time. OpenAI's o1 series popularized this idea, and the DeepSeek-R1 paper describes how to get there with RL.

The Ingredients

Most reasoning models share five building blocks:

• A strong base model: RL amplifies skills the model already shows occasionally. It rarely creates them from nothing.
• Chain-of-thought format: the model writes out its reasoning, then the final answer.
• Verifiable rewards: programs check final answers for math, code, and similar tasks.
• Supervised stages: curated examples stabilize format and style.
• Distillation: strong reasoning models teach smaller ones.

The interesting part lies in how these fit together.

The DeepSeek-R1 Pipeline, Stage by Stage

DeepSeek's R1 paper gives the clearest public recipe, so let's walk through it.

Starting Point: R1-Zero (Pure RL)

DeepSeek-R1-Zero applied large-scale reinforcement learning directly to a base model, with no supervised fine-tuning at all. The reward came from simple checks of correctness and format. Reasoning behaviors emerged during training: longer chains of thought, self-verification, and reflection on earlier steps.

The result proved a point: reasoning can emerge from RL alone. It also came with problems. Outputs read poorly, and the model sometimes mixed languages mid-chain.

Stage 1: Cold-Start SFT

To fix R1-Zero's messiness, the team started R1 differently. They collected a small set of high-quality long chain-of-thought examples, on the order of thousands, and fine-tuned the base model on them. This stage isn't there to teach reasoning. It anchors the format and tone so the RL phase starts from a stable, readable place.

Stage 2: Reasoning-Oriented RL

Next came large-scale RL on reasoning tasks, using the same style of training as R1-Zero, with GRPO as the algorithm. Rewards came from verifiers: did the final answer match, and did the output follow the required format? Reports also describe a reward that encouraged consistent language to cut down on language mixing.

This stage supplies most of the capability gain. The model learns to spend more tokens thinking and to check its own work.

Stage 3: Rejection Sampling and SFT

Near convergence, the team used the RL checkpoint to generate many solutions per prompt, then kept only the good ones. They combined those reasoning traces with general-purpose supervised data (writing, question answering, and similar tasks) to build a large new SFT dataset, reported at roughly 800 thousand samples across the pipeline. Fine-tuning on it makes the model's best behavior reliable and restores general abilities.

def build_sft_data(prompts, generate, verify, n=8):
    data = []
    for p in prompts:
        samples = [generate(p) for _ in range(n)]
        good = [s for s in samples if verify(p, s)]
        if good:
            data.append({"prompt": p, "response": min(good, key=len)})
    return data

This sketch keeps the shortest correct response per prompt, which also nudges the model away from needless rambling.

Stage 4: Final RL

A last RL phase aligns the full model across a wider range of prompts, including general helpfulness and safety, so the finished model behaves well outside math and code. Reports describe this pipeline as two RL stages interleaved with two SFT stages.

Distillation: The Surprise

Here's one of the most practical findings. DeepSeek took the curated traces and fine-tuned smaller dense models (Qwen-based and Llama-based, from 1.5 billion to 70 billion parameters) using supervised fine-tuning only, with no RL on the small models.

The distilled models performed strongly. Reports say the 7B distilled model beat a non-reasoning GPT-4o version on reasoning benchmarks, and the 32B and 70B versions surpassed OpenAI's o1-mini on several benchmarks. The paper also compared distillation against running RL directly on a small model, and distillation won. Applying RL on top of the distilled students added only marginal gains in the project's reports.

The practical lesson: if a strong reasoning teacher exists, distill from it first. RL on small models costs a lot and often loses to simply learning from a better model's traces.

Figure 1: The chain-of-thought format is a contract — write the steps, then commit to an answer

The Reward Design Behind It

Reasoning RL leans on verifiable rewards:

• Accuracy rewards: does the final answer match? Do the unit tests pass?
• Format rewards: does the response separate reasoning from the final answer in the expected structure?

def reasoning_reward(completion, gold):
    return 1.0 * accuracy(completion, gold) + 0.1 * format_ok(completion)

The R1 report describes avoiding neural reward models for the reasoning RL stage, because learned rewards invite reward hacking at scale. Process reward models, which score each step, offer denser feedback, and recent research shows they can behave like fluency detectors under optimization pressure. Outcome-based checks remain the safer default — the RLVR guide covers verifier design in depth.

What RL Teaches, and What It Doesn't

RL gave reasoning models their signature behaviors: longer chains, self-checking, and backtracking. But research raises a sober question about where those abilities come from.

A widely discussed 2025 study found that RLVR-trained models beat their base models at small numbers of samples, while base models caught up or won when researchers sampled many attempts. The authors conclude that RLVR mostly sharpens sampling toward reasoning paths the base model already contains, while distillation from a stronger teacher can add genuinely new patterns. Other researchers dispute parts of that framing, arguing that pass@k can mislead when base models reach right answers through flawed reasoning. The debate remains open.

The practical takeaway is the same either way: pick the strongest base you can, distill when a teacher exists, and use RL to sharpen and stabilize.

Test-Time Compute: The Other Half

Training produces a model that can think. Inference decides how much it does think. Several techniques spend extra compute at test time:

• Longer chains of thought: more reasoning tokens often improve accuracy on hard problems.
• Majority voting: sample several solutions, and return the most common answer.
• Best-of-N with a verifier: generate candidates, score them, and return the best.
• Budget controls: cap or extend thinking length per request.

More thinking costs more money and adds latency. Production systems usually let users or routers choose how much reasoning each request deserves.

A Practical Recipe for Smaller Teams

You don't need frontier compute to apply these ideas. A realistic path:

1. Choose a strong base or instruct model that solves some of your target problems.
2. Gather or generate reasoning traces from the best teacher you can legally and practically use, then filter them with a verifier.
3. Fine-tune with SFT on those filtered traces (distillation).
4. Run RLVR with GRPO on tasks with reliable verifiers, using prompts of medium difficulty.
5. Evaluate on held-out problems with pass@1 and larger-k pass@k, and audit reasoning quality by hand.
6. Control length so the model doesn't overthink easy questions.

Check the licenses and terms for any teacher model's outputs before you use them as training data.

Pitfalls to Expect

• Overthinking: models can write long chains for trivial questions. Use length penalties and routing.
• Language mixing and unreadable chains: R1-Zero showed this, and cold-start data helps fix it.
• Reward hacking: policies exploit weak verifiers. Test them adversarially, and sandbox code execution.
• Contamination: benchmark problems may leak into training data. Hold out fresh problems.
• Unfaithful chains of thought: studies suggest a model's written reasoning doesn't always reflect what actually drove its answer, so treat chains as useful signals, not guaranteed explanations.
• Cost: generating many long samples per prompt dominates training expense.

Common Mistakes

• Running RL on a weak base — no successful samples means no signal.
• Skipping distillation when a strong teacher exists — it's usually the cheaper win.
• Training on prompts that are too easy or too hard — reward groups go uniform.
• Ignoring readability — pure RL chains can turn into gibberish.
• Judging by one benchmark — use several, plus held-out data from the evaluation toolkit.
• Forgetting inference cost — longer thinking means a bigger bill.
• Skipping verifier stress tests — a loophole turns into a training objective.
• Trusting reasoning chains as explanations — audit outputs, not narratives.

Your Quick-Start Checklist

1. Pick a base model that sometimes solves your target tasks.
2. Build a verifier and test it for loopholes.
3. Collect or generate filtered reasoning traces, and try SFT first.
4. Add RLVR with GRPO or a similar algorithm on medium-difficulty prompts.
5. Use a small cold-start set if outputs turn unreadable.
6. Evaluate on held-out problems and audit samples.
7. Set length and cost controls for deployment.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What makes a model a reasoning model, and what are the building blocks?

A reasoning model generates a long internal chain of thought before it commits to a final answer. Instead of jumping from question to answer in a few tokens, it spends hundreds or thousands of tokens working through the problem — writing out steps, checking them, backing up when something looks wrong, and only then answering. That behavior doesn't come free with a bigger model; someone trained it in. The shift opens a new scaling dimension: test-time compute. Traditional scaling improved models by training bigger networks on more data, while reasoning models also improve by thinking longer at inference time — OpenAI's o1 series popularized the idea, and the DeepSeek-R1 paper describes how to get there with RL. Most reasoning models share five building blocks: a strong base model (RL amplifies skills the model already shows occasionally and rarely creates them from nothing), a chain-of-thought format where the model writes out reasoning then the final answer, verifiable rewards where programs check final answers for math and code, supervised stages with curated examples that stabilize format and style, and distillation where strong reasoning models teach smaller ones. The interesting part lies in how these fit together into a pipeline, not in any one ingredient.

What are the stages of the DeepSeek-R1 training pipeline?

DeepSeek's R1 paper gives the clearest public recipe. Starting point, R1-Zero (pure RL): large-scale reinforcement learning applied directly to a base model with no supervised fine-tuning at all, with rewards from simple checks of correctness and format — reasoning behaviors emerged during training, including longer chains of thought, self-verification, and reflection on earlier steps, but outputs read poorly and the model sometimes mixed languages mid-chain. Stage 1, cold-start SFT: to fix that messiness the team collected a small set of high-quality long chain-of-thought examples (on the order of thousands) and fine-tuned the base model on them — this stage isn't there to teach reasoning but to anchor format and tone so RL starts from a stable, readable place. Stage 2, reasoning-oriented RL: large-scale RL on reasoning tasks using GRPO, with rewards from verifiers (did the final answer match, did the output follow the required format) plus a language-consistency reward to cut mixing — this stage supplies most of the capability gain, teaching the model to spend more tokens thinking and to check its own work. Stage 3, rejection sampling and SFT: near convergence the RL checkpoint generates many solutions per prompt, only the good ones are kept, and those traces are combined with general-purpose supervised data to build a large new SFT dataset (roughly 800 thousand samples across the pipeline), making the best behavior reliable and restoring general abilities. Stage 4, final RL: a last RL phase aligns the full model across a wider prompt range including general helpfulness and safety — two RL stages interleaved with two SFT stages.

Why does distillation beat running RL on small models?

One of the most practical findings in the R1 work: DeepSeek took the curated traces and fine-tuned smaller dense models (Qwen-based and Llama-based, from 1.5 billion to 70 billion parameters) with supervised fine-tuning only, no RL on the small models. The distilled models performed strongly — reports say the 7B distilled model beat a non-reasoning GPT-4o version on reasoning benchmarks, and the 32B and 70B versions surpassed OpenAI's o1-mini on several benchmarks. The paper also compared distillation directly against running RL on a small model, and distillation won; applying RL on top of the distilled students added only marginal gains in the project's reports. The practical lesson: if a strong reasoning teacher exists, distill from it first, because RL on small models costs a lot and often loses to simply learning from a better model's traces. The same logic drives the reward design: accuracy rewards check whether the final answer matches or the unit tests pass, format rewards check that reasoning is separated from the final answer, and the R1 report describes avoiding neural reward models for the reasoning RL stage because learned rewards invite reward hacking at scale. Process reward models, which score each step, offer denser feedback, but recent research shows they can behave like fluency detectors under optimization pressure, so outcome-based checks remain the safer default.

How does test-time compute fit in, and what should smaller teams watch out for?

Training produces a model that can think; inference decides how much it does think. Techniques that spend extra compute at test time include longer chains of thought (more reasoning tokens often improve accuracy on hard problems), majority voting (sample several solutions and return the most common answer), best-of-N with a verifier (generate candidates, score them, return the best), and budget controls that cap or extend thinking length per request. More thinking costs more money and adds latency, so production systems usually let users or routers choose how much reasoning each request deserves. A realistic recipe for smaller teams: choose a strong base or instruct model that solves some of your target problems; gather or generate reasoning traces from the best teacher you can legally and practically use, then filter them with a verifier; fine-tune with SFT on those filtered traces (distillation); run RLVR with GRPO on tasks with reliable verifiers using prompts of medium difficulty; evaluate on held-out problems with pass@1 and larger-k pass@k and audit reasoning quality by hand; and control length so the model doesn't overthink easy questions. Check the licenses and terms for any teacher model's outputs before using them as training data. Pitfalls to expect: overthinking on trivial questions, language mixing and unreadable chains, reward hacking against weak verifiers, benchmark contamination, unfaithful chains of thought (a model's written reasoning doesn't always reflect what actually drove its answer, so treat chains as useful signals rather than guaranteed explanations), and the cost of generating many long samples per prompt, which dominates training expense.

Wrapping This Up

Reasoning models come from a pipeline, not a single trick: a strong base model, chain-of-thought training, verifiable-reward RL, curated SFT stages, and distillation to spread the results. DeepSeek-R1 showed that reasoning can emerge from RL on verifiable tasks, and that a four-stage recipe makes it reliable and readable.

If you're building your own, start with the strongest base and teacher you can access, distill before you reach for RL, and treat every verifier as something a clever policy will try to break. The thinking looks magical, and the recipe stays refreshingly practical.

Related Articles

• GRPO Explained: Group Relative Policy Optimization for Reasoning Models
• RLVR: Reinforcement Learning with Verifiable Rewards Explained
• Reward Models Explained: Teaching a Model What "Good" Looks Like
• LLM Evaluation: Testing and Measuring Outputs the Right Way