Research map / Reinforcement learning / Visual and representation RL
BlendRL: A Framework for Merging Symbolic and Neural Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BlendRL, a neuro-symbolic RL framework that blends a neural CNN policy with an LLM-generated differentiable logic policy via a learned blending function, trained end-to-end with PPO-style actor-critic on Atari games.
Problem
RL policies are either opaque neural networks or symbolic logic policies that need expert-provided predicates and rules; neither handles both high-level reasoning and low-level reactive control well, and symbolic policies struggle with fine motor actions.
Approach
Runs a neural policy on pixels and a differentiable forward-reasoning logic policy on object-centric states in parallel, combining their action distributions with a state-dependent blending weight; LLM (GPT-4o) generates the action rules, predicates, valuation functions and blending rules, and a hybrid value function with an entropy-style regularizer on the blend weight is used.
Key results
- BlendRL achieves human normalized scores of 474.4 on Kangaroo, 94.7 on Seaquest and 48.4 on DonkeyKong, outperforming PPO, DQN, NUDGE, SCoBots, INTERPRETER and INSIGHT in the reported table.
- On HackAtari modified environments (enemies removed, relocated ladders), BlendRL keeps positive returns (e.g. 187 on Kangaroo, 113 on Seaquest, 263 on DonkeyKong) while neural PPO scores 0.0 on all three.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BlendRL instead of SCoBots | Atari policy learning · Kangaroo | human normalized score: 149.7 -> 474.4 | Helped |
| Use BlendRL instead of DQN | Atari policy learning · Seaquest | human normalized score: 61.3 -> 94.7 | Helped |
| Use BlendRL instead of PPO | Atari policy learning · DonkeyKong | human normalized score: 28.4 -> 48.4 | Helped |
| Use BlendRL instead of Neural PPO on modified environments | Atari policies · Modified Kangaroo | return: 0.0_{\pm 0.0} -> 187_{\pm 120} | Helped |
| Use BlendRL instead of Neural PPO on modified environments | Atari policies · Modified Seaquest | return: 0.0_{\pm 0.0} -> 113_{\pm106} | Helped |
| Use BlendRL instead of Neural PPO on modified environments | Atari policies · Modified DonkeyKong | return: 0.0_{\pm 0.0} -> 263_{\pm124} | Helped |
| Use a symbolic blending module instead of a neural blending module (ablation) | BlendRL · Kangaroo and Seaquest | performance | Helped |
Surprising findings
- Neural PPO agents collapse to suboptimal policies (e.g. surfacing without divers, never refilling oxygen in Seaquest), which the logic module helps avoid.
- A symbolic blending module performed better than a neural blending module on Kangaroo and Seaquest.
Limitations
- Relies on LLM-generated predicates, rules and valuation functions, with some rules and weights tuned to maximize performance.
- Evaluated only on three Atari games with a relatively small number of seeds (3 for curves, 10 for robustness).
Experimental setup
Datasets: Atari Learning Environment (Kangaroo, Seaquest, DonkeyKong) HackAtari modified environments
Benchmarks: Atari Kangaroo Atari Seaquest Atari DonkeyKong HackAtari robustness variants
Metrics: human normalized score raw episodic return robustness return
Baselines: PPO DQN NLRL NUDGE SCoBots INTERPRETER INSIGHT
Builds on: NUDGE Neural Logic Reinforcement Learning PPO differentiable forward reasoning SCoBots
Closest papers by meaning
- End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations (ICML 2024 · similarity 0.92)
- VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning (ICLR 2025 · similarity 0.88)
- Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization (ICLR 2025 · similarity 0.87)
- Lifting Traces to Logic: Programmatic Skill Induction with Neuro-Symbolic Learning for Long-Horizon Agentic Tasks (ICML 2026 · similarity 0.86)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.86)
- Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.