Research map / Reinforcement learning / Visual and representation RL
Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SYMPOL, which integrates the GradTree differentiable axis-aligned decision tree into PPO (and A2C) so the interpretable tree policy is optimized directly on-policy, avoiding post-hoc distillation information loss.
Problem
Interpretable symbolic RL policies such as decision trees are typically obtained by post-hoc distillation or discretization of neural or soft-tree policies, which causes a mismatch and information loss between the trained and deployed policy; learning hard axis-aligned trees directly in on-policy RL
Approach
Directly optimize a hard, axis-aligned decision tree policy with policy gradients by using GradTree's arithmetic dense DT formulation with a straight-through estimator, plus a dynamic rollout buffer and batch size, weight decay on feature-index and leaf parameters, and a separate NN critic.
Key results
- SYMPOL achieves 500 on CartPole and -57 on LunarLander, outperforming SA-DT (d=5) at -197 and D-SDT at -221 among interpretable methods.
- Cohen's d between validation and test reward is -0.019 for SYMPOL versus 3.449 for SA-DT (d=5), 2.527 for SA-DT (d=8), and 3.126 for D-SDT.
- SYMPOL trees average 50.5 nodes after pruning, compared with 60.3 for SA-DT (d=5) and 291.6 for SA-DT (d=8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| directly optimize the interpretable tree policy instead of distilling or discretizing a trained policy | SYMPOL versus SA-DT and D-SDT · Control benchmarks (mean) | Cohen's D: -0.019 versus 3.449, 2.527, and 3.126 | Helped |
| use a separate neural-network critic architecture rather than the actor's architecture (ablation) | SYMPOL · Control environments | normalized mean reward: 0.080 -> 0.988 | Helped |
| use a dynamic rollout buffer rather than a static rollout buffer (ablation) | SYMPOL · Control environments | normalized mean reward: 0.598 -> 0.988 | Helped |
| increase batch size dynamically rather than using a static batch size (ablation) | SYMPOL · Control environments | normalized mean reward: 0.372 -> 0.988 | Helped |
| use AdamW with weight decay rather than Adam without weight decay (ablation) | SYMPOL · Control environments | normalized mean reward: 0.865 -> 0.988 | Helped |
| use direct tree optimization rather than post-hoc distillation or discretization | SYMPOL versus SA-DT and D-SDT · Control benchmarks: CartPole, Acrobot, LunarLander, MountainCarContinuous, Pendulum | average undiscounted cumulative test reward | Mixed |
Surprising findings
- Using A2C instead of PPO widens the gap in favor of SYMPOL; on MountainCarContinuous only SYMPOL achieves a positive reward, while full-complexity MLP and SDT do not.
- The authors attribute the poor distillation results partly to the instability of the full-complexity policy rather than the distillation procedure itself, though they note further experiments would be needed.
Limitations
- Evaluated only on small control and MiniGrid environments with small trees (depth 7).
- Relies on a fully-grown tree with post-hoc pruning, and early training can be unstable (e.g., MountainCar).
- Restricted to on-policy actor-critic methods; off-policy extensions (SAC, AWR) left to future work.
Experimental setup
Datasets: CartPole Acrobot LunarLander MountainCarContinuous Pendulum MiniGrid Empty-Random MiniGrid DoorKey MiniGrid LavaGap MiniGrid DistShift
Benchmarks: CartPole Acrobot LunarLander MountainCarContinuous Pendulum MiniGrid Empty-Random MiniGrid DoorKey MiniGrid LavaGap MiniGrid DistShift
Metrics: average undiscounted cumulative test reward Cohen's d (information loss) tree size (nodes) runtime
Baselines: SA-DT D-SDT VIPER MLP SDT
Builds on: GradTree PPO VIPER DAGGER
Closest papers by meaning
- In Search of Trees: Decision-Tree Policy Synthesis for Black-Box Systems via Search (AAAI 2025 · similarity 0.88)
- SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes (NeurIPS 2025 · similarity 0.88)
- ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies (NeurIPS 2026 · similarity 0.88)
- BlendRL: A Framework for Merging Symbolic and Neural Policy Learning (ICLR 2025 · similarity 0.87)
- RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space (NeurIPS 2024 · similarity 0.87)
- End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.