Research map / Reinforcement learning / Visual and representation RL
End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INSIGHT, a neuro-symbolic RL framework that distills vision foundation models (FastSAM+DeAOT) into a perception module for object coordinates, jointly fine-tunes it with an EQL symbolic policy via neural guidance, and uses GPT-4 to generate textual explanations.
Problem
Neuro-symbolic RL from visual observations cannot refine structured object states with rewards because prior perception (e.g., SPACE) is too costly to fine-tune, and symbolic policies are hard for non-expert users to interpret.
Approach
Distill segmentation and tracking from vision foundation models into a lightweight perception module that predicts object coordinates, train the EQL symbolic actor jointly with a neural actor via a neural guidance cross-entropy loss so rewards refine the perception module end-to-end, and prompt GPT-4 with concept grounding and chain-of-thought rules to explain policies and decisions.
Key results
- INSIGHT reaches 20.9 on Pong, 3828 on BeamRider, and 409.6 on Breakout, matching Neural and outperforming NS-RL baselines CGP, DiffSES, DSP and NUDGE on all nine Atari tasks.
- On MetaDrive INSIGHT's success rate at 5M steps is 0.63 versus 0.49 for Neural.
- Inference time per state is 2 ms, the same as Neural and roughly 25x faster than SPACE-Neural (50 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · Atari tasks | task return | Helped |
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · Pong | F-MAE (×100): 3.1\pm0.3 -> 1.3\pm0.9 | Helped |
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · BeamRider | F-MAE (×100): 4.9\pm0.1 -> 3.2\pm0.3 | Helped |
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · Enduro | F-MAE (×100): 12.2\pm0.5 -> 6.7\pm0.3 | Helped |
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · SpaceInvaders | F-MAE (×100): 11.1\pm0.4 -> 8.1\pm0.9 | Helped |
| fine-tune the perception module with reward signals instead of freezing it (ablation) | INSIGHT · Freeway | F-MAE (×100): 18.7\pm0.8 -> 14.8\pm1.6 | Helped |
| pre-train the perception module (ablation) | INSIGHT · Atari tasks | task return | Helped |
| add neural guidance to symbolic policy learning (ablation) | INSIGHT · Atari tasks | task return | Helped |
| add neural guidance to symbolic policy learning (ablation) | INSIGHT · Pong | F-MAE (×100): 1.6\pm0.9 -> 1.3\pm0.9 | Helped |
| add neural guidance to symbolic policy learning (ablation) | INSIGHT · BeamRider | F-MAE (×100): 5.9\pm0 -> 3.2\pm0.3 | Helped |
| add neural guidance to symbolic policy learning (ablation) | INSIGHT · Enduro | F-MAE (×100): 13.4\pm0.2 -> 6.7\pm0.3 | Helped |
| add neural guidance to symbolic policy learning (ablation) | INSIGHT · SpaceInvaders | F-MAE (×100): 11.1\pm0.4 -> 8.1\pm0.9 | Helped |
Surprising findings
- Optimizing the perception loss and neural guidance loss for multiple PPO iterations led to inferior performance, so they are updated only in the last iteration.
- Coor-Neural is outperformed by the pixel-based Neural agent on six tasks, suggesting object coordinates alone are not expressive enough for online policy learning.
Limitations
- EQL network cannot express logical operations needed by some reasoning tasks.
- Quantitative evaluation of the generated explanations is left for future work.
- Pretraining the perception module adds a complex two-stage training protocol; MsPacman F-score is low due to many spurious objects.
Experimental setup
Datasets: Atari games (Pong, BeamRider, Enduro, Qbert, SpaceInvaders, Seaquest, Breakout, Freeway, MsPacman) MetaDrive OCAtari
Benchmarks: Atari (nine games) MetaDrive
Metrics: test return success rate MAE F-MAE inference time F-score
Baselines: Neural CGP DiffSES DSP NUDGE SA-Neural SPACE-Neural CleanRL
Builds on: PPO EQL SPACE FastSAM DeAOT
Closest papers by meaning
- BlendRL: A Framework for Merging Symbolic and Neural Policy Learning (ICLR 2025 · similarity 0.92)
- Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization (ICLR 2025 · similarity 0.86)
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning (ICLR 2025 · similarity 0.86)
- Deciphering Raw Data in Neuro-Symbolic Learning with Provable Guarantees (AAAI 2024 · similarity 0.85)
- Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning (ICML 2024 · similarity 0.84)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.