stable-baselines3

Stable Baselines3 - Production-Grade Reinforcement Learning Algorithms Library

Skills Overview

Stable Baselines3 is an implementation library of reinforcement learning algorithms based on PyTorch. It provides production-ready implementations of PPO, SAC, DQN, TD3, DDPG, A2C, and others. It uses a clean, scikit-learn–like API, and is suitable for standard RL experiments, rapid prototype development, and deployment in production environments.

Use Cases

1. Standard Reinforcement Learning Experiments

Suitable for single-agent reinforcement learning research and experiments in Gymnasium environments. It offers complete training templates, monitoring callbacks, and evaluation tools. It supports a wide range of standard benchmark environments, from CartPole to MuJoCo, making it an ideal choice for academic research and algorithm validation.

2. Rapid Prototype Development

For scenarios that require quickly validating RL ideas, Stable Baselines3 provides ready-to-use algorithm implementations and detailed documentation, including custom environment templates, training scripts, and evaluation tools. Developers can start training experiments within minutes without having to implement complex algorithm details from scratch.

3. Single-Agent Production Applications

Suitable for single-agent applications that require reliable RL algorithm implementations. The code has been thoroughly tested and includes production-grade features such as model persistence, vectorized environment support, and TensorBoard integration. For multi-agent systems or scenarios requiring high-performance parallel training, pufferlib is recommended.

Core Features

1. Unified Training Interface for Multiple Algorithms

Provides popular reinforcement learning algorithms such as PPO, SAC, DQN, TD3, DDPG, and A2C. All algorithms share the same training API, including the learn() method, model saving/loading, and parameter management. It supports multiple policy types, including CNN policies for image observations and MultiInput policies for dictionary observations, covering different task requirements.

2. Environment Integration and Validation

Supports full integration with Gymnasium environments and provides a check_env() tool to verify the correctness of custom environments. Supports vectorized environment implementations for parallel training, including SubprocVecEnv for multi-process execution and DummyVecEnv for single-process execution. Includes environment checkers to ensure that observation spaces, action spaces, and the step function comply with SB3 standards.

3. Training Monitoring and Evaluation

Includes a rich callback system supporting model evaluation during training, checkpoint saving, early stopping strategies, and progress monitoring. Provides the evaluate_policy() tool for quantitative evaluation of a trained agent and supports recording training videos via VecVideoRecorder. Integrates TensorBoard for real-time monitoring of training metrics, making debugging and performance analysis easier.

Frequently Asked Questions

Which reinforcement learning algorithms does Stable Baselines3 support?

It supports algorithms such as PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), DQN (Deep Q-Network), TD3 (Twin Delayed DDPG), DDPG (Deep Deterministic Policy Gradient), and A2C (Advantage Actor-Critic). In addition, the SB3-Contrib package provides experimental algorithms such as MaskablePPO, CrossQ, QR-DQN, and RecurrentPPO.

How do I train a model using stable-baselines3?

First create a Gymnasium environment, then initialize the algorithm model, and finally call the learn() method to train. For example: model = PPO("MlpPolicy", env, verbose=1); model.learn(total_timesteps=10000). After training, save the model using model.save(). Load the saved model with PPO.load(). For detailed training templates, refer to scripts/train_rl_agent.py.

What are the differences between stable-baselines3 and pufferlib?

Stable Baselines3 focuses on standard, single-agent reinforcement learning implementations, offering comprehensive documentation and thoroughly validated algorithms that are suitable for quick experiments and production applications. pufferlib is designed for high-performance parallel training, multi-agent systems, and custom vectorized environment optimization. It is more advantageous for scenarios that require large-scale parallel or distributed training. If your needs are standard RL experiments, SB3 is the better choice.