Hugging Face TRL Tutorial: Preference Optimization in Python

Contents

Classification loss, preference data, no reward model

Supervised fine-tuning teaches a model what to say. Preference optimization teaches it which of two answers people like better. If your model is fluent but tone-deaf, rambling or unsafe, that second step is often the one you need.

This tutorial covers Hugging Face's TRL library and its DPOTrainer: the data format, the quickstart, the knobs that matter, and the metrics to watch. It's the hands-on companion to our DPO explainer. A caution up front: the TRL snippets here come from the library's documentation and are untested against a live install, so confirm the current docs for your version; the DPO math was checked with a small toy implementation, which is what makes the numbers below meaningful.

What DPO Does

The DPO paper's abstract introduces "a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form," so you can "solve the standard RLHF problem with only a simple classification loss." In plain terms:

• Classic RLHF: train a reward model on human preferences, then use reinforcement learning to optimize against it.
• DPO: skip the reward model and the RL loop. Train directly on pairs of (preferred, rejected) responses with a classification-style loss.

The authors call it "stable, performant, and computationally lightweight" and report that it matches or improves on PPO-based RLHF on the tasks they tested, including sentiment control, summarization and single-turn dialogue. Those are the authors' results. The abstract doesn't tell you how well they carry over to other settings.

The Data: Prompt, Chosen, Rejected

TRL's docs say DPO needs a preference dataset, in standard or conversational format. The recommended form has an explicit prompt:

# Standard format (from the docs)
preference_example = {"prompt": "The sky is", "chosen": " blue.", "rejected": " green."}

# Conversational format (from the docs)
preference_example = {
    "prompt":   [{"role": "user", "content": "What color is the sky?"}],
    "chosen":   [{"role": "assistant", "content": "It is blue."}],
    "rejected": [{"role": "assistant", "content": "It is green."}],
}

With conversational data, the trainer applies the chat template for you. An implicit-prompt format also works, where the prompt is repeated inside chosen and rejected, but the docs recommend the explicit form. To convert another dataset, the docs use dataset.map(...).

Where Do Pairs Come From?

• Human annotators ranking two outputs.
• Production feedback, like thumbs up or down and user edits.
• A judge model ranking samples — check its biases first, since position and length bias are known problems with LLM judges.

Data quality dominates everything else. Noisy or inconsistent preferences teach the model noise. The same labeler-agreement discipline from our human evaluation guide applies here.

The Quickstart

From the TRL docs:

from trl import DPOTrainer
from datasets import load_dataset

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()

That's the whole docs example. Swap in your own model and dataset — and check the current docs for version-specific changes before you copy it.

The Reference Model

DPO needs a reference policy to measure how far you've drifted. The docs say that if you don't pass ref_model, the trainer "will automatically use the initial policy corresponding to model, i.e. the model state before DPO training starts." Useful options from the same page:

• precompute_ref_log_probs (default False): computes reference log-probabilities up front so the reference model needn't stay in memory.
• sync_ref_model (default False): periodically syncs the reference with the policy, following TR-DPO, with ref_model_sync_steps (default 512) and ref_model_mixup_alpha (default 0.6).
• PEFT: pass peft_config=LoraConfig() to train adapters. The docs suggest a learning rate around 1e-5 for adapters, and list constraints (for example, sync_ref_model=True can't be combined with precompute_ref_log_probs=True).

The Loss, and What Beta Means

The docs give the loss as:

L = −E[ log σ( β ( log(π_θ(y⁺|x) / π_ref(y⁺|x)) − log(π_θ(y⁻|x) / π_ref(y⁻|x)) ) ) ]

Read it like this. For each pair, compare how much the policy has increased the probability of the chosen answer, relative to the reference, against how much it increased the rejected one. The loss pushes the first gap above the second.

Two properties are worth checking on a toy policy (a softmax over eight candidate responses):

1. When the policy equals the reference, every margin is zero and the loss is exactly log 2 = 0.693, for any beta. If your first logged loss is far from 0.693, something is off with your setup. That's a property of the formula, not a promise about every trainer configuration.
2. The analytic gradient matched finite differences to better than 1e-8.

What Beta Changes in Practice

The docs say beta defaults to 0.1 and describe it as controlling "the deviation from the reference model. Higher beta means less deviation from the reference model."

At the optimum, training on the full preference distribution converges to the closed form from the DPO paper: the optimal policy is proportional to the reference policy times exp(reward/β). So the theory agrees with the docs: at the optimum, smaller beta lets the policy move further. KL from the reference was 1.782 at beta = 0.1 and 0.111 at beta = 2.0.

But that's the optimum. With 100 noisy preference pairs and a fixed budget of 300 gradient steps, the picture changed:

Beta	KL from reference	Training accuracy
0.05	0.087	0.77
0.10	0.270	0.77
0.50	0.715	0.77
2.00	0.156	0.77

The movement was not monotone in beta. A likely explanation, untested, is that tiny beta barely moves in a short run (the gradient scales with beta) and large beta fits the pairs quickly and stops. Training accuracy was 0.77 for all of them, capped by the noisy labels. The practical lesson: don't pick beta by name alone. Track the KL divergence from the reference and evaluate on held-out preferences.

Figure 1: The four TRL reward metrics tell you about the data; only your eyes tell you about the model

The Metrics TRL Logs

The docs define four to watch:

• rewards/chosen: the average implicit reward for chosen completions, β × log(π_θ/π_ref).
• rewards/rejected: the same for rejected ones.
• rewards/margins: the average gap between them.
• rewards/accuracies: the share of pairs where the chosen completion's implicit reward is higher.

The margin is just chosen minus rejected, and accuracy is the share of positive margins. A healthy run usually shows margins and accuracies rising together. Treat that as a rule of thumb, not a guarantee. A rising accuracy on training pairs can mean memorization, so also check held-out accuracy.

The docs also list logps/chosen, logps/rejected, grad_norm and others. Watch logps/chosen too. A drop in the chosen response's absolute likelihood is a known failure mode of the DPO objective, so treat it as a caution to monitor rather than a documented promise of the trainer page.

Loss Variants

The docs list many loss_type options beyond the default "sigmoid", including "hinge", "ipo", "robust" (for noisy preferences, with label_smoothing in [0.0, 0.5)), "bco_pair", "sft" and "sigmoid_norm" (a length-normalized SimPO-style loss). You can combine them:

training_args = DPOConfig(
    loss_type=["sigmoid", "bco_pair", "sft"],
    loss_weights=[0.8, 0.2, 1.0]
)

That example is from the docs. Start with the default, and only try others when you can name a specific problem, like noisy labels or length bias, that the variant targets — the DPO family section maps variants to the problems they fix.

A Sensible Workflow

1. Start from an SFT model that already follows your format.
2. Build 1,000 or more clean preference pairs. (That count is a rule of thumb, not a documented number.)
3. Hold out a preference test set that you never train on.
4. Run a short pilot with the defaults, and confirm the first loss is near 0.693.
5. Watch margins, accuracies, KL and sample outputs. Read generations by hand.
6. Compare against the SFT model on your own evaluation set, not just the preference metric.
7. Change one thing at a time, such as beta, learning rate or data.

Common Mistakes

• Preference noise. Contradictory labels cap your accuracy.
• Length bias. If chosen answers are always longer, the model learns "longer," not "better."
• No held-out check. Training accuracy rising isn't evidence of improvement.
• Reading outputs too late. Metrics can look fine while outputs get odd.
• Chasing the benchmark. A better preference score doesn't mean a better product.
• Skipping the pilot. The log-2 starting loss catches setup bugs in one step.
• Ignoring logps/chosen. A falling chosen likelihood is the quiet failure mode.
• Tuning before cleaning. The same discipline as unit-testing outputs: fix the data first, then the knobs.

The highest-value habit is simple: read twenty generations from the tuned model next to the original every time you change anything. Concrete samples teach more than any metric list.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What data format does TRL's DPOTrainer expect?

DPO needs a preference dataset, in standard or conversational format. In the standard format each record has an explicit prompt plus a chosen and a rejected completion, as in prompt "The sky is", chosen " blue.", rejected " green.". In the conversational format the prompt holds message lists (a user message asking what color the sky is) and chosen and rejected each hold an assistant message — the trainer applies the chat template for you. An implicit-prompt format also works, where the prompt is repeated inside chosen and rejected, but the docs recommend the explicit form, and converting another dataset is a matter of dataset.map. Where do the pairs come from? Human annotators ranking two outputs, production feedback such as thumbs up or down and user edits, or a judge model ranking samples — and with judges, check the biases first, since position and length bias are known problems. Data quality dominates everything else: noisy or inconsistent preferences teach the model noise, and contradictory labels cap the accuracy you can reach.

What does beta control in DPO, and how should you pick it?

The docs say beta defaults to 0.1 and describe it as controlling the deviation from the reference model: higher beta means less deviation from the reference. The loss itself compares how much the policy has increased the probability of the chosen answer, relative to the reference, against how much it increased the rejected one — pushing the first gap above the second. Two sanity checks are worth running with a toy policy (a softmax over eight candidate responses). First, when the policy equals the reference every margin is zero and the loss is exactly log 2 = 0.693 for any beta, so if your first logged loss is far from 0.693 something is off with your setup — that is a property of the formula, not a promise about every trainer configuration. Second, the analytic gradient should match finite differences to better than 1e-8. At the optimum, training on the full preference distribution converges to the closed form from the DPO paper, where the optimal policy is proportional to the reference policy times exp(reward divided by beta), so smaller beta lets the policy move further. But that is the optimum: with 100 noisy preference pairs and a fixed budget of 300 gradient steps, the observed KL divergence from the reference was not monotone in beta — 0.087 at beta 0.05, 0.270 at 0.10, 0.715 at 0.50 and 0.156 at 2.00, all at 0.77 training accuracy, which was capped by the noisy labels. A likely explanation is that tiny beta barely moves in a short run because the gradient scales with beta, while large beta fits the pairs quickly and stops. The practical lesson: don't pick beta by name alone — track KL divergence from the reference and evaluate on held-out preferences.

Which metrics should you watch while training with TRL?

The docs define four reward metrics to watch: rewards/chosen (the average implicit reward for chosen completions, beta times log of the policy/reference ratio), rewards/rejected (the same for rejected ones), rewards/margins (the average gap between them) and rewards/accuracies (the share of pairs where the chosen completion's implicit reward is higher). The margin is simply chosen minus rejected, and accuracy is the share of positive margins, so a healthy run usually shows margins and accuracies rising together — treat that as a rule of thumb, not a guarantee, because rising accuracy on training pairs can mean memorization, which is why you also check held-out accuracy. The docs also list logps/chosen, logps/rejected, grad_norm and others. Watch logps/chosen too: a drop in the chosen response's absolute likelihood is a known failure mode (likelihood displacement), though that specific point is a caution rather than a documented promise of the trainer page. Alongside the numbers, read generations by hand and compare against the SFT model on your own evaluation set rather than only the preference metric.

Which TRL loss variants exist, and when should you use them?

The docs list many loss_type options beyond the default sigmoid, including hinge, ipo, robust (for noisy preferences, with label_smoothing in the range 0.0 to 0.5), bco_pair, sft and sigmoid_norm (a length-normalized SimPO-style loss), and you can combine several with weights — for example loss_type of sigmoid, bco_pair and sft with weights 0.8, 0.2 and 1.0. Start with the default, and only try others when you can name a specific problem that the variant targets, such as noisy labels or length bias. A sensible workflow overall: start from an SFT model that already follows your format; build 1,000 or more clean preference pairs (a rule of thumb rather than a documented number); hold out a preference test set you never train on; run a short pilot with the defaults and confirm the first loss is near 0.693; watch margins, accuracies, KL and sample outputs, reading generations by hand; compare against the SFT model on your own evaluation set; and change one thing at a time, such as beta, learning rate or data. Common mistakes: preference noise, length bias (if chosen answers are always longer, the model learns longer rather than better), no held-out check, reading outputs too late, and chasing the benchmark when a better preference score doesn't mean a better product.

Wrapping This Up

DPO turns preference pairs into a classification-style loss, with a frozen reference model keeping the policy from drifting too far. TRL's DPOTrainer wraps this, with prompt/chosen/rejected data, a default beta of 0.1 and metrics for rewards, margins and accuracy.

The things that decide your results are data quality, beta, and whether you check held-out behavior rather than training metrics. Next step: build 200 careful preference pairs, run the quickstart pattern on a small model, and read the outputs. Then decide whether to scale up.

Related Articles

• DPO Explained: Preference Tuning Without Reinforcement Learning
• Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement
• Bias in LLM-as-a-Judge: Position, Verbosity, and Self-Preference
• DeepEval Tutorial: Unit Test Your LLM Outputs in Python