Research map / Reinforcement learning / Visual and representation RL
Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Successive Concept Bottleneck Agents (SCoBots) that chain object-property and relational concept bottlenecks with a PPO-trained ReLU policy distilled into a decision tree, enabling inspection, pruning and reward shaping, and uncovering a goal misalignment in Pong.
Problem
Deep RL agents suffer from goal misalignment, reward sparsity and credit assignment issues, and their black-box policies prevent domain experts from inspecting and revising them.
Approach
Decompose the RL policy into successive concept bottlenecks: object properties extracted from frames, then human-readable relational concepts (distances, speed, intersections), then a distilled decision-tree action selector, allowing expert pruning of concepts and actions plus concept-based expert rewards.
Key results
- SCoBots perform at least on par with deep PPO agents on all 9 games and slightly outperform them on 5 (Asterix, Boxing, Kangaroo, Seaquest, Tennis).
- Unguided SCoBots and deep agents drop to random-level HN scores on Pong NoEnemy, revealing a misalignment where the agent tracks the enemy paddle instead of the ball; pruning the enemy restores play.
- SCoBots train in on average 7.5 hours versus 10.8 hours for deep agents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCoBots instead of deep PPO agents | Atari agents · Asterix, Boxing, Kangaroo, Seaquest, Tennis | human-normalized score | Helped |
| Remove the enemy object from the concepts available to the policy | Pong SCoBots · Pong and NoEnemy | human-normalized score | Helped |
| Add a reward based on distance from the mother kangaroo to the joey | Guided SCoBots · Kangaroo | level completion | Helped |
| Add flag-progression and speed rewards | Guided SCoBots · Skiing | human score | Helped |
| Add a reward inversely proportional to paddle-ball distance | SCoBots · Pong | winning-shot occurrences: every $\sim \!820$, multiplying by $2.7$ their occurrences | Helped |
| Add object-detection noise (5% misdetection probability and Gaussian noise with 3-pixel standard deviation) (ablation) | NN-SCoBots · Atari games, especially Kangaroo | human-normalized score | Mixed |
| Remove explicit relations (ablation) | Decision-tree SCoBots · Atari games | performance | Hurt |
| Remove explicit relations (ablation) | NN-SCoBots · Atari games | performance | No clear effect |
Surprising findings
- Deep and SCoBot Pong agents that look well-behaved under importance maps fail completely when the enemy is hidden, revealing a previously unknown shortcut that post-hoc explanations missed.
- Ablating relational concepts hurts decision-tree SCoBots on 6 of 9 games but barely affects neural object-centric agents.
Limitations
- Relies on OCAtari quasi-perfect object extraction from RAM; imperfect detection causes drops (e.g., Kangaroo).
- Object-centric representation does not cover non-object concepts such as mazes (e.g., MsPacman).
- Training time advantage disappears in environments with many objects (Kangaroo) due to sequential CPU computation.
Experimental setup
Datasets: Atari Learning Environments (9 games) HackAtari Pong variants (NoEnemy, LazyEnemy)
Benchmarks: Atari Learning Environments (9 games: Asterix, Bowling, Boxing, Freeway, Kangaroo, Pong, Seaquest, Skiing, Tennis) HackAtari Pong NoEnemy and LazyEnemy
Metrics: human-normalized score training time return
Baselines: PPO with CNN (deep agents) NN-SCoBots (neural object-centric baseline) PPO-v4 original results from Schulman et al.
Builds on: concept bottleneck models PPO VIPER OCAtari HackAtari
Closest papers by meaning
- LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization (ICML 2025 · similarity 0.84)
- Interpreting Neural Combinatorial Optimization via Evolving Programmatic Bottlenecks (NeurIPS 2026 · similarity 0.84)
- Causally Reliable Concept Bottleneck Models (NeurIPS 2025 · similarity 0.84)
- Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions (NeurIPS 2025 · similarity 0.84)
- Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.