Research map / Reinforcement learning / Visual and representation RL
Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a goal-reducer VAE trained with loop-removal sampling (filtration-based) to generate subgoals, integrates it with DQL and SAC, runs a standalone recursive variant, and compares its activations with human fMRI in a treasure-hunting task.
Problem
Goal-directed RL is inefficient when goals change because the state-goal space is huge and experience is not reusable across goals; classical planners need predefined global structure.
Approach
A goal-reducer network learns s,g -> subgoal associations from replay trajectories after removing loops via a persistent-homology-inspired filtration radius, requiring no prior environment distance knowledge or Q-function; it accelerates DQL/SAC or runs standalone with a local policy.
Key results
- In the four-room navigation task, DQL with goal-reducer (DRL+GR) converges faster than plain DQL across grid sizes 13-21.
- The standalone goal-reducer (GR) with local policy outperforms both GR+DRL and plain DRL in a 19x19 four-room maze.
- RSA shows goal-reducer hidden layer (Φ) correlates with parts of vmPFC and putamen/basal ganglia regions in 24 human subjects' fMRI data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add goal-reducer to DQL | DQL · Four-room navigation | convergence time | Helped |
| Add goal-reducer to SAC | SAC · Robot arm reach | performance | Helped |
| Use standalone goal-reducer with a local policy and remove Bellman equation training | DQL + goal-reducer · 19x19 four-room navigation | performance | Helped |
| Train the goal-reducer with loop-removal sampling instead of trajectory sampling (ablation) | Goal-reducer · Geometric random graph and four-room gridworld | training loss | Helped |
| Train the goal-reducer with trajectory sampling instead of random sampling (ablation) | Goal-reducer · Geometric random graph and four-room gridworld | training loss | Helped |
| Train the goal-reducer with loop-removal sampling instead of random sampling (ablation) | Goal-reducer · Geometric random graph and four-room gridworld | Optimality | Helped |
| Recursively reduce goals for three iterations (ablation) | Goal-reducer trained with loop-removal sampling · Geometric random graph and four-room gridworld | Optimality | Helped |
| Increase four-room navigation maze sizes while using goal-reducer (ablation) | DRL+GR · Four-room navigation | performance relative to DRL | Helped |
Surprising findings
- The standalone goal-reducer without Bellman iteration outperforms goal-reducer-augmented DQL in the navigation task.
Limitations
- Loop-removal sampling lacks a clear biological correspondence.
- Filtration radius must be tuned by grid search per environment.
- Brain-data ROI comparisons are descriptive; fMRI data only shared upon request.
Experimental setup
Datasets: four-room gridworld (Minigrid-based) panda-gym robot arm reach treasure hunting fMRI dataset (Zhang et al. 2023 / ref 54)
Benchmarks: four-room navigation panda-gym robot arm reach 19x19 four-room maze treasure hunting fMRI RSA
Metrics: convergence time / episode steps Optimality Equidex RSA z-values training loss
Baselines: DQL (DRL) SAC Trajectory sampling Random sampling GR+DRL
Builds on: Deep Q-learning Soft Actor-Critic Hindsight Experience Replay variational autoencoder persistent homology
Closest papers by meaning
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.86)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.85)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.85)
- Learning to Reach Goals via Diffusion (ICML 2024 · similarity 0.85)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.