Research map / Reinforcement learning / Policy gradient and actor-critic
Gradient Boosting Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GBRL, a framework that trains gradient-boosted tree ensembles as RL actor-critic function approximators via incremental functional-gradient updates, and shows advantages over MLPs on structured/categorical and OOD-robustness tasks while lagging on Atari RAM.
Problem
Neural networks struggle with structured, categorical observations and generalize poorly out-of-distribution in RL, while gradient boosting trees (GBT) excel on tabular supervised data but existing GBT libraries are designed for static datasets and are too slow for RL's incremental updates.
Approach
Reinterpret the GBT ensemble as the policy and value parameterization, fitting a new tree at each iteration to the functional policy gradient and value gradient, with a shared actor-critic tree structure (each leaf outputs both policy and value) and a CUDA-accelerated implementation for incremental boosting.
Key results
- GBRL outperforms MLP PPO in MiniGrid tasks such as PutNear, FourRooms and Fetch, and matches MLP on classic control (e.g., CartPole 500 for both under PPO).
- On Atari RAM, GBRL underperforms NN in most games (e.g., PPO Alien 1163.86 vs 1555.32), attributed to unstructured features.
- In incremental learning, GBRL trains 1000 boosting iterations in 5.47s vs 36.17s for CatBoost and 67.11s for XGBoost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GBRL instead of CatBoost or XGBoost for incremental tree learning | GBT libraries · Incremental learning, 1000 iterations | training time (sec): $36.17 \pm 2.77$; $67.11 \pm 0.70$ -> $\mathbf{5.47} \pm \mathbf{1.19}$ | Helped |
| Use GBRL PPO instead of NN PPO | PPO · MountainCar | average episode reward: $-115.53 \pm 1.39$ -> $-110.55 \pm 15.60$ | Helped |
| Use GBRL PPO instead of NN PPO | PPO · Football Academy, Run to score w/ keeper | average episode reward: $1.13 \pm 0.06$ -> $1.95 \pm 0.02$ | Helped |
| Use GBRL PPO instead of NN PPO | PPO · Atari RAM | average episode reward | Hurt |
| Use GBRL instead of NN | RL policies · MiniGrid | episodic reward | Helped |
| Use GBRL instead of NN | RL policies · Football | episodic reward | Helped |
| Use GBRL instead of NN | RL policies · Classic-control tasks | episodic reward | No clear effect |
| Use GBRL instead of NN | Variable isolation policies · Variable isolation task | average episodic reward | Helped |
| Use GBRL instead of NN | MiniGrid policies · Spurious-correlations configurations | episodic reward | Helped |
| Add irrelevant objects to observations | GBRL and NN policies · MiniGrid Fetch | episodic reward | Mixed |
| Add random Gaussian noise to evaluation states | GBRL rather than NN policies · Classic-control | episodic reward | Helped |
| Vary the policy or value learning rate (ablation) | GBRL PPO · CartPole-v1 | average episodic reward | Mixed |
Surprising findings
- Standard MLP PPO solves Atari-RAM and most Football scenarios better than GBRL, so GBRL's advantages are domain-specific.
- CatBoost and XGBoost were intractable for RL in the reported experiments, with CatBoost lacking GPU support for custom losses.
Limitations
- Ensemble size grows unboundedly with continual tree construction, raising memory and compute costs.
- GBTs are non-differentiable, so off-policy continuous control methods like DDPG and SAC are not directly supported.
- Tree-based models struggle with unstructured high-dimensional inputs such as Atari RAM.
Experimental setup
Datasets: Gymnasium Classic-Control LunarLander-v2 Google Research Football Atari RAM MiniGrid Variable isolation environment (custom)
Benchmarks: Gymnasium Classic-Control LunarLander-v2 Google Research Football Atari RAM MiniGrid Variable isolation
Metrics: average episodic reward over final 100 episodes normalized score SHAP values wall-clock training time (incremental learning)
Baselines: MLP PPO MLP A2C MLP AWR CatBoost XGBoost NPPG
Builds on: PPO A2C AWR XGBoost Stable-Baselines3
Closest papers by meaning
- Reinforcement Learning Guided Semi-Supervised Learning (NeurIPS 2024 · similarity 0.87)
- Efficient Reinforcement Learning by Discovering Neural Pathways (NeurIPS 2024 · similarity 0.86)
- TorchRL: A data-driven decision-making library for PyTorch (ICLR 2024 · similarity 0.86)
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- SAPG: Split and Aggregate Policy Gradients (ICML 2024 · similarity 0.86)
- GFlowNet Training by Policy Gradients (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.