Research map / Reinforcement learning / Visual and representation RL

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills small oblique-tree rule-based surrogates with linear action heads from deep RL policies in continuous action spaces, and uses them for fidelity, performance, and policy-improvement analysis.

Problem

Interpretable surrogate models for deep RL policies with continuous actions over mixed continuous-discrete state spaces are lacking; existing tree and rule methods handle only discrete actions or axis-aligned splits and yield large models.

Approach

Learns oblique decision trees via a three-stage split search (random hyperplane initialization through PCA-reduced space, differential-evolution refinement, backward elimination), fits per-leaf linear action models, then merges adjacent leaves into compact DNF-style rules, embedded in a DAgger loop.

Key results

• Orcaid achieves higher reward ratio than DT and RuleFit at comparable or smaller model size across environments, with Cubist competitive but larger.
• Mirrored-rule enforcement finds policy improvements in Orcaid models at a higher rate than other surrogates; e.g., Lunar Lander Orcaid depth-1 rate 0-89% vs DT up to 100% at larger size.
• LLM-judge scores favor Orcaid over DT in 100% of comparisons for 7 of 9 environments (mean scores e.g. 4.3 vs 2.1 on MC).

Surprising findings

• Surrogate rules can improve the original RL policy when enforced in mirrored regions, showing the RL policy has exploitable weaknesses.
• Fidelity (low MSE) does not always translate into higher reward due to model drift.

Limitations

• Only deterministic policies are modeled; stochastic policies need extension.
• Scaling beyond 17-dim states and 6-dim actions may require dimensionality reduction.
• Interpretability assessed by LLM-as-a-judge rather than human study; Hopper and Half Cheetah reward ratio is lower than DT in some cases.

Experimental setup

Datasets: MuJoCo and Gym control tasks (MC, P, LL, IP, IDP, R, S, H, HC)

Benchmarks: Mountain Car Pendulum Lunar Lander Inverted Pendulum Inverted Double Pendulum Reacher Swimmer Hopper Half Cheetah

Metrics: reward ratio to RL policy test MSE (fidelity) model size (parameter count) policy improvement rate via mirrored rules LLM-judged explainability score

Baselines: DT (CART) Cubist RuleFit VIPER-style DT with DAgger

Builds on: VIPER CART DAgger RuleFit LEGIBLE

Closest papers by meaning

• Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization (ICLR 2025 · similarity 0.88)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.87)
• RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space (NeurIPS 2024 · similarity 0.87)
• Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
• RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation (ICML 2024 · similarity 0.87)