Sam Austinon November 7, 202616 min read

Tianshou vs ElegantRL vs CleanRL vs Pearl: Which RL Library Repo Should You Use?

Tianshou vs ElegantRL vs CleanRL vs Pearl: Which RL Library Repo Should You Use?
Contents

Figure 1: Tianshou vs ElegantRL vs CleanRL vs Pearl — four libraries promise "clean," "elegant," or "production-ready" RL. Which one actually delivers?

You've decided to stop writing PPO from scratch. Four repos keep showing up: Tianshou, ElegantRL, CleanRL, and Pearl. Each one promises "clean," "elegant," or "production-ready" RL, and each README makes the others look underpowered. So which one deserves your next three weeks?

I cannot claim I have trained agents in all four, so I will give you my honest analysis from their docs and READMEs. I also searched for current details, and I need to flag a problem upfront: most of what I found was README text, mirrors, and forks, not fresh maintenance data. I could not confirm recent commit activity for any of these, so you will need to check that yourself. IMO, README comparison tables are marketing, and every library in this article publishes one that flatters itself. Ready to sort it out?

What Are We Comparing?

These four libraries solve overlapping problems with very different philosophies:

  • Tianshou: a modular PyTorch platform with high-level and low-level APIs
  • ElegantRL: a lightweight, massively parallel framework aimed at cloud-scale training
  • CleanRL: single-file implementations you read and modify, not import
  • Pearl: Meta's modular agent library aimed at production features like safety and dynamic action spaces

Ever picked a library for its feature list and then spent a week fighting its abstractions? Philosophy matters more than features here.

Tianshou: The Modular Generalist

Tianshou describes itself as a reinforcement learning platform based on pure PyTorch and Gymnasium, supporting online and offline RL, multi-agent RL, and model-based algorithms. It also offers two interface levels: a high-level interface for ease of use and a procedural interface for maximum control. That dual design suits both beginners and algorithm developers. GitHub

What I like:

  • Broad algorithm coverage, including PPO, SAC, TD3, offline methods like CQL and BCQ, and imitation learning such as GAIL, based on its README list
  • Gymnasium support: the project dropped support for the old gym API and recommends Gymnasium
  • Tests that check reproducibility: its README says tests fail if agents don't reach a consistent performance level on limited epochs

What to check:

  • Release lag: one README version warns that the latest PyPI release is currently far behind the master branch. Check whether that still applies before you decide between pip install tianshou and installing from source
  • API changes: the same README warns of breaking changes during a major transition, so pin your version
  • Reading difficulty: modular frameworks hide algorithms behind abstractions, so debugging takes longer

I would pick Tianshou when I need multiple algorithm families, offline RL, or multi-agent support in one library.

ElegantRL: The Lightweight, Parallel Option

ElegantRL positions itself as a lightweight, massively parallel deep reinforcement learning framework optimized for efficiency, stability, and cloud-native deployment, implemented in PyTorch. It comes from the AI4Finance community, and it serves other projects such as FinRL and FinRL-Meta. FinRL itself supports ElegantRL, RLlib, and Stable-Baselines3 as backends, which tells you where its audience lives: financial RL.

What I like:

  • Compact core: the README says the core code runs under 1,000 lines, with a HelloWorld tutorial for beginners
  • Algorithm coverage: it lists DDPG, TD3, SAC, PPO, and REDQ for continuous actions, and DQN variants for discrete actions
  • Multi-GPU training: one mirror claims it can use 0 to 8 GPUs for training

What to check:

  • Self-reported stability claims: the authors say ElegantRL shows much more stable performance than Stable-Baselines3. That is the authors' claim, so reproduce it on your problem before you believe it
  • Environment compatibility: several tutorials I found used OpenAI Gym and PyBullet Gym, and some date from 2021. Test Gymnasium support before you commit
  • Maintenance signals: one repo-tracking site lists it at about 4,175 stars and calls it six years old, so confirm recent commits yourself
  • Documentation age: the FAQ I found carries a 2021 date

I would choose ElegantRL for financial RL experiments through FinRL, or when parallel throughput matters more than ecosystem polish.

CleanRL: Read It, Don't Import It

CleanRL takes the opposite approach from the other three. Its documentation says plainly that it is not a modular library and is not meant to be imported. Instead, it provides high-quality single-file implementations with research-friendly features, covering PPO, DQN, C51, DDPG, TD3, SAC, and PPG.

What I like:

  • Readable algorithms: one file per algorithm, so you can follow every line
  • Experiment tracking: it integrates with TensorBoard and Weights & Biases
  • Transparent benchmarks: it participates in Open RL Benchmark, which holds tracked experiments from popular DRL libraries
  • Academic backing: it has a JMLR paper
  • Offline spin-off: CORL implements offline RL algorithms in CleanRL style

What to check: you copy and modify files, so you own your fork's maintenance. Updating ten diverged copies of ppo.py gets old fast. It also lacks the modularity you would want for a production service.

I would choose CleanRL for learning, for research baselines, and for any project where you want to modify the algorithm itself.

Pearl: Meta's Production-Oriented Agent Library

Pearl calls itself a production-ready reinforcement learning AI agent library from the Applied Reinforcement Learning team at Meta. It targets features that production teams care about, like environments with limited observability, sparse feedback, and high stochasticity. Its README also notes that Pearl marks the sunset of ReAgent, Meta's earlier RL project.

What I like:

  • Production-oriented features: the README highlights dynamic action spaces, offline RL, intelligent exploration, safe decision making, history summarization, and contextual bandits
  • Modular agent design: you combine policy learners, replay buffers, and exploration modules
  • State dicts like PyTorch modules: the README says you can save and load agent state with torch.save and torch.load
  • Real-world framing: it targets recommender systems, auction bidding, and creative selection use cases

What to check:

  • Maturity signals: the README announces a v0.1 beta release, and the repo page I found shows no published releases. Meta's team presented it at NeurIPS 2023, so confirm whether anyone still actively develops it
  • Install path: the README tells you to clone the repo and run pip install -e ., not install from PyPI
  • Self-flattering tables: its README includes a feature table where Pearl scores check marks and rivals like Tianshou score X marks on agent modularity. Treat any such table as the authors' view

I would choose Pearl for exploring contextual bandits, safe RL, dynamic action spaces, or recommender and bidding problems, after confirming it is still maintained.

Quick Comparison

Here is my fast take on who fits where:

  • Tianshou: best all-around modular library, with offline and multi-agent support
  • ElegantRL: best for parallel training and FinRL-style financial experiments
  • CleanRL: best for learning, research baselines, and algorithm hacking
  • Pearl: best for production-style features like safety, bandits, and dynamic actions

And a rougher split by use case:

  • I want to learn PPO line by line: CleanRL
  • I want many algorithms in one API: Tianshou
  • I want a trading-agent pipeline: ElegantRL through FinRL
  • I want contextual bandits or recommender-style RL: Pearl
  • I want the most mature ecosystem and docs: consider Stable-Baselines3 instead, as the previous article in this series discussed

How to Choose Your Library

Skip the feature tables for a minute. Start with three honest questions.

What Are You Optimizing For?

Learning and research clarity point toward CleanRL. Breadth and flexibility point toward Tianshou. Parallel throughput points toward ElegantRL. Production-style features point toward Pearl.

How Much Maintenance Risk Can You Take?

All four carry risk, since I could not verify recent commit activity for any of them. Check issues, releases, and Gymnasium compatibility before you build on one. Pin exact versions and keep a fallback.

Does Your Problem Match the Library's Home Turf?

ElegantRL grew up around finance, and Pearl around recommendation and bidding. A routing or scheduling problem may fit Tianshou or CleanRL better, and Stable-Baselines3 with SB3-Contrib's action masking worked well in this series' operations tutorials.

How to Run a Fair Trial

Don't trust README benchmarks, including mine. Follow this protocol:

  1. Pick one real task: your own Gymnasium environment, not just CartPole
  2. Install each library in a clean environment and record the install pain
  3. Train PPO or SAC with several seeds in each, using matched settings
  4. Compare paired results, with confidence intervals and a strong non-RL baseline
  5. Time everything: setup, training, debugging, and iteration speed
  6. Test one custom change, like a new reward or masking rule, and see how hard it is
  7. Check the issue tracker for unanswered questions

Did you notice how few of those steps involve reading a feature table?

Common Mistakes to Avoid

  • Trusting README comparisons. Every library flatters itself.
  • Ignoring Gym versus Gymnasium. Older tutorials and code may break on new environments.
  • Skipping version pinning. RL libraries change fast and break quietly.
  • Choosing by stars. One repo-tracking site's numbers say little about whether anyone still fixes bugs.
  • Over-abstracting too early. A modular framework helps at scale and hurts when you just want to understand PPO.
  • Comparing libraries on one seed. Differences between libraries often hide inside seed variance.
  • Forgetting action masking. Operations problems usually need it, so check each library's support first.

Final Thoughts

There is no single winner. CleanRL teaches you the algorithms, Tianshou gives you breadth, ElegantRL gives you parallel throughput and a finance community, and Pearl targets production-style problems if it stays alive. Check maintenance, test Gymnasium compatibility, and run your own benchmark.

So pick the library that matches your problem this week, then train it against a simple baseline on three seeds. If it learns and you understand why, you found your tool. If your curve stays flat, congratulations: you just learned that library choice rarely causes the problem, and your reward function probably did. :)

Frequently Asked Questions

Which RL library is best for beginners?

Stable-Baselines3 (SB3) is still the best starting point. If you want to learn from code, CleanRL is the best for understanding algorithms. Tianshou is good if you need many algorithms in one API.

Which RL library is best for action masking in operations problems?

SB3-Contrib's MaskablePPO on top of Stable-Baselines3 is the most mature option for action masking. Tianshou also supports action masking but is less battle-tested.

Which library should I use for multi-agent reinforcement learning?

PettingZoo provides the standard API. For training, RLlib and PufferLib handle distributed training. Tianshou has built-in multi-agent support but is less battle-tested than RLlib.

Which library is best for financial RL?

ElegantRL with FinRL is purpose-built for financial RL. It has built-in environments for portfolio optimization, trading, and portfolio management.

How do I know if an RL repository is actually maintained?

Check for recent commits (within 30-60 days), active issue responses, recent PyPI releases, and whether it supports the current Gymnasium/PyTorch versions. Abandoned repos often break within months.

Sources

  • [1] Tianshou Documentation, Tianshou: A Reinforcement Learning Platform — modular RL platform: tianshou.readthedocs.io
  • [2] AI4Finance Foundation, ElegantRL Documentation — lightweight parallel RL: elegantrl.readthedocs.io
  • [3] V. V. V. et al., CleanRL: High-quality single-file implementations of deep RL algorithms — documentation and benchmarking: vwxyzjn.github.io/cleanrl
  • [4] Meta AI Research, Pearl: A Production-Ready Reinforcement Learning Agent Library — arXiv preprint: arxiv.org/abs/2312.00673
  • [5] Farama Foundation, PettingZoo: Multi-agent RL environments — multi-agent API: pettingzoo.farama.org

What are You Looking For?

esc