Sam Austinon November 7, 202616 min read

GitHub Repos for Training LLMs with Reinforcement Learning: verl and Its Alternatives Compared

GitHub Repos for Training LLMs with Reinforcement Learning: verl and Its Alternatives Compared
Contents

Figure 1: GitHub Repos for Training LLMs with RL — you want to train a reasoning model with GRPO. Which framework actually works at your scale?

You want to train a reasoning model with GRPO. You open GitHub and find verl, TRL, OpenRLHF, NeMo-RL, SkyRL, AReaL, ROLL, and a dozen more, each claiming speed, flexibility, or production readiness. Pick the wrong one, and you burn a week of GPU time debugging memory errors instead of studying rewards.

I cannot claim I have trained models in all of these, so I will give you my honest analysis from their docs, papers, and community write-ups. I also need to flag a problem: this space changes monthly, and many comparisons come from the frameworks' own authors. IMO, the best way to read this article is as a map, then verify the terrain yourself. Ready to find your match?

Why LLM RL Frameworks Differ From Classic RL Libraries

If you followed earlier articles in this series, you used small environments and tiny networks. LLM RL changes the engineering problem completely. Each training step needs a generation phase (the model writes answers), a scoring phase (a reward model or verifier grades them), and a learning phase (the optimizer updates a huge model).

Those phases stress different systems:

  • Rollout engines: vLLM or SGLang generate tokens fast
  • Training engines: FSDP or Megatron-LM shard the model across GPUs
  • Orchestration: Ray or similar tools coordinate everything
  • Rewards: verifiers, code sandboxes, or reward models score outputs

One survey notes that established frameworks like OpenRLHF and verl rely on Ray-based distributed computing to manage training workflows. So the real question is not "Which algorithm?" but "Which system handles my scale and my reward setup?" arxiv

Ever watched a GPU sit idle while another one runs out of memory? That is the problem these repos try to solve.

verl: The Research-Scale Default

verl describes itself as a flexible, efficient, and production-ready RL training library for large language models, and it is the open-source release of the HybridFlow paper from ByteDance's Seed team and collaborators. The project also migrated to the verl-project organization in January 2026, so check the official repository's current location and avoid the many forks that search results surface.

What I like:

  • Broad algorithm support: its README lists PPO, GRPO, GSPO, ReMax, REINFORCE++, RLOO, PRIME, DAPO, Dr.GRPO, and others
  • Swappable backends: it supports FSDP, FSDP2, and Megatron-LM for training, and vLLM, SGLang, and Hugging Face Transformers for rollout generation
  • Verifiable rewards: it supports model-based and function-based rewards for math, coding, and similar tasks
  • Multi-turn and multimodal work: it lists vision-language models and multi-turn rollouts with tool calling
  • Hardware range: it supports NVIDIA, AMD, and Ascend hardware
  • Scale: it claims support for models up to 671B parameters with expert parallelism across hundreds of GPUs

What to check:

  • Breaking changes: the README links a list of breaking changes since v0.4, so pin your version and read release notes
  • Dependency traps: the README says it supports vLLM 0.8.2 or newer with FSDP and warns against vLLM 0.7.x because of bugs that cause out-of-memory errors. Version mismatches cost people days
  • Architecture in motion: the roadmap items include async and off-policy architecture, so expect change
  • Learning curve: flexible systems demand configuration work. A third-party guide describes it as the place where the latest research lands, while calling OpenRLHF the cleaner codebase to read

I would pick verl when I need current algorithms (GRPO, DAPO-style variants), multi-node scale, or agentic multi-turn RL.

TRL: The Easiest On-Ramp

Hugging Face's TRL gives you trainers for PPO, DPO, GRPO, and other methods inside the familiar Transformers ecosystem. A survey table summarizes it as a Hugging Face toolkit for post-training transformers with algorithms such as PPO and DPO. One community guide calls it the best starting point for people new to RL code and recommends reading its PPO trainer before touching any Ray-based framework.

What I like:

  • Low friction: it fits notebooks and single-node experiments
  • Familiar APIs: if you know Transformers, you already know half of it
  • Readable trainers: you can follow the algorithm in the source

What to check:

  • Scale limits: the OpenRLHF paper argues that TRL-style tools provide accessible implementations but often lack sophisticated orchestration and struggle with inference optimization. That is a competitor's view, so test it on your workload
  • Version drift: trainer APIs change, so pin versions

I would pick TRL for learning, small models, and quick GRPO or DPO experiments.

OpenRLHF: Readable and Ray-Based

OpenRLHF markets itself in its paper as an easy-to-use, scalable, high-performance framework covering RLHF and RLVR. It uses Ray for distributed orchestration and aims for a simpler architecture than industrial frameworks. A community comparison calls it the easiest production-grade RL codebase to read.

What I like:

  • Readability: the code reads end to end more easily than most large frameworks, according to that community guide
  • Production orientation: it targets real distributed training
  • Active research use: institutions have adopted it, per its authors

What to check: the paper claims OpenRLHF consistently trains faster than verl across tested model sizes and context lengths in long chain-of-thought RLVR. That comes from the authors' own benchmark on their chosen settings. Verl's maintainers may report different results on different hardware and configurations, so run your own comparison before you trust either side.

I would pick OpenRLHF when I want a Ray-based system I can actually read and modify.

NeMo-RL: The NVIDIA-Scale Option

NVIDIA's NeMo-RL targets very large clusters and tight integration with Megatron-based stacks. One community guide recommends it for thousand-GPU NVIDIA-only setups, while noting that its codebase couples tightly to NVIDIA frameworks. Treat that as one blogger's opinion and read NVIDIA's own docs for current capabilities.

I would consider it if I already run NVIDIA's training stack at large scale. Smaller teams rarely need it.

The Agentic and Asynchronous Newcomers

A newer wave focuses on multi-turn agents, tool use, and asynchronous training:

  • SkyRL: a survey describes it as a modular full-stack RL framework for agentic training. Another paper notes it offers broad tool support, including bash terminals and SQL executors, but needs complex containerized environments that add deployment overhead
  • AReaL: the same survey labels it a fully asynchronous RL framework
  • ROLL: listed as user-friendly large-scale RL
  • Prime-RL and NeMo-RL: one environment-framework comparison notes that a few frameworks ship their own trainer, like Prime RL, NeMo RL, and SkyRL, while others need adapters to plug into external loops like TRL
  • RAGEN, rLLM, Search-R1, Agent Lightning, and others: research-oriented projects for agent and tool-use RL

That same environment comparison warns that there is no standard protocol yet for how LLMs interact with RL environments, and that its framework details may have changed since May 2026. I would add my own warning: fast-moving research repos sometimes stop updating within months. Check recent commits before you build on one.

Quick Comparison

Here is my fast take on who fits where:

  • TRL: best for learning, notebooks, and small-scale experiments
  • OpenRLHF: best for readable, Ray-based production-style training
  • verl: best for current research algorithms, flexible backends, and multi-node scale
  • NeMo-RL: best for very large NVIDIA-centric clusters
  • SkyRL: best for agentic multi-tool training, if you can handle container complexity
  • AReaL: best for exploring fully asynchronous RL
  • ROLL, Prime-RL, and others: best for teams that match their specific designs

How to Choose the Right Framework

Skip the benchmark claims for a minute. Start with three honest questions.

What Scale Do You Need?

One GPU or a small node points toward TRL. A handful of nodes with serious models points toward verl or OpenRLHF. Thousands of GPUs with Megatron points toward NeMo-RL or verl with Megatron. Be honest about your cluster, not your ambitions.

What Are You Training On?

Single-turn math or code with verifiable rewards works well in verl, TRL, and OpenRLHF. Multi-turn tool use and agents push you toward verl's multi-turn features, SkyRL, or similar agentic frameworks. Rewards from sandboxes and external tools add engineering work no matter which you pick.

Who Will Maintain It?

A framework that moves fast breaks your pipeline fast. Pin your versions, track release notes, and budget time for upgrades. A readable codebase helps when you hit a bug at 2 a.m.

How to Run a Fair Trial

Don't trust speed tables, including the ones in the papers I cited. Follow this protocol:

  1. Pick a small model first: a 0.5B to 1.5B model on a task with a verifiable reward, like GSM8K-style math
  2. Install each framework in a clean environment with pinned versions, and record setup time
  3. Match settings: same model, data, batch size, sequence length, and number of samples per prompt
  4. Measure three things: tokens per second, GPU memory, and final task accuracy
  5. Run several seeds, because RL variance can swamp small differences
  6. Test one custom change, like a new reward function, and time how long it takes
  7. Scale up only after the small run works

Did you notice how few of those steps involve reading a README?

Common Mistakes to Avoid

  • Trusting author benchmarks. Every framework's paper shows it winning, so reproduce on your hardware
  • Starting at scale. Debug on a small model before you rent a cluster
  • Ignoring version pinning. Mismatched vLLM, PyTorch, or driver versions cause silent failures
  • Using unofficial forks. Search results surface many forks and mirrors, so clone from the official organization
  • Weak rewards. A leaky verifier teaches the model to cheat. Test your reward function on adversarial outputs
  • Skipping evaluation. Track held-out accuracy, not just training reward
  • Chasing every new algorithm. GRPO variants multiply constantly, so compare against a plain baseline first

Final Thoughts

There is no universal winner. TRL teaches you the loop, OpenRLHF gives you a readable distributed system, and verl gives you the broadest research toolkit and scale. NeMo-RL and the agentic newcomers fit specific needs. Match the framework to your scale, your reward setup, and your tolerance for churn.

So pick one small task this week, a 1B-parameter model and a verifiable reward, and run it in TRL and one other framework. If both learn, you found your tool by evidence instead of marketing. If the reward curve climbs while held-out accuracy does not, congratulations: you just met reward hacking, and no framework fixes that for you.

Frequently Asked Questions

Which LLM RL framework should I start with as a beginner?

TRL is the easiest on-ramp. It works inside the Hugging Face ecosystem you likely already know, supports single-node training, and has readable trainers for PPO, DPO, and GRPO.

Which framework should I use for multi-node, large-scale LLM RL training?

verl with FSDP or Megatron-LM backends. It supports multi-node training, expert parallelism, and models up to 671B parameters. OpenRLHF is a readable Ray-based alternative.

Which framework is best for GRPO training of reasoning models?

verl and TRL both support GRPO. verl has broader algorithm support (GRPO, GSPO, DAPO, PRIME, Dr.GRPO) and multi-turn support. TRL is simpler for single-node experiments.

Do I need Ray for LLM RL training?

verl, OpenRLHF, and SkyRL use Ray for orchestration. TRL can run without Ray on a single node. NeMo-RL uses NVIDIA's stack instead. Your cluster setup determines the choice.

How do I avoid breaking changes in these fast-moving repos?

Pin exact versions (e.g., verl==0.4.x, vllm==0.8.2), read release notes before updating, test on small models first, and track the official organization repos, not forks.

Sources

What are You Looking For?

esc