Research map / Reinforcement learning / Visual and representation RL
Predictive auxiliary objectives in deep RL mimic learning in the brain
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that predictive auxiliary losses (a latent transition model with contrastive negative sampling) in a double-DQN gridworld agent help prevent representational collapse and improve transfer, and that the resulting representations mirror hippocampal, visual cortex and striatal-like findings.
Problem
How predictive auxiliary objectives shape representation learning across modules of a deep RL system, and whether these representational changes mimic those observed in the brain (hippocampus, visual cortex, striatum).
Approach
Trains a small double-DQN agent end-to-end with a self-predictive latent transition loss (a positive sampling term with horizon gamma plus an exponential negative sampling term) and treats encoder, Q-network and transition network as analogs of visual cortex, striatum and hippocampus for comparison with neural data.
Key results
- Predictive auxiliary loss lets the agent learn the foraging task with smaller latent sizes than MF-only or negative-sampling-only models.
- Longer predictive horizons (gamma up to 0.8) transfer better to new goal locations, with the benefit stronger at larger latent sizes.
- Predictive objectives hurt transfer when transition structure is scrambled, with performance in task B lower than negative-sampling-only at epsilon=0.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the full predictive auxiliary loss instead of using no auxiliary loss | Double DQN · 8×8 gridworld | training speed to maximum episode score | Helped |
| Add the full predictive auxiliary loss rather than negative sampling alone | Double DQN · Gridworld with a smaller latent bottleneck | episode score | Helped |
| Add the full predictive auxiliary loss rather than negative sampling alone | Double DQN · CIFAR gridworld | episode score | Helped |
| Add the full predictive auxiliary loss | Double DQN · Gridworld with stochastic transitions | episode score | No clear effect |
| Use longer-horizon prediction (larger γ) rather than one-step prediction (ablation) | MF + predictive auxiliary loss · Goal transfer in gridworld | episode score | Helped |
| Use predictive auxiliary loss rather than negative sampling alone | Double DQN · Transfer with scrambled transition structure | episode score | Hurt |
| Remove the predictive auxiliary loss (ablation) | Double DQN with encoder · Gridworld representation learning | fraction of silent encoder units | Hurt |
| Remove the predictive auxiliary loss (ablation) | Double DQN encoder · Preference-swap transition learning | change in encoder stimulus responses | Hurt |
| Add predictive auxiliary loss to the memory-equipped model | Double DQN · Alternating-T maze | population response splitting | Helped |
Surprising findings
- Predictive auxiliary objectives reduce transfer performance when the transition structure changes between tasks, underperforming the negative-sampling-only model.
- The benefit of predictive auxiliary objectives vanishes as transition stochasticity increases.
Limitations
- Tasks are simple gridworld environments with limited complexity.
- Limited diversity of auxiliary objectives tested.
- Value learning network not examined against striatum data.
Experimental setup
Datasets: gridworld (8x8 arena, shuffled and CIFAR-10 observation variants)
Benchmarks: gridworld foraging CIFAR gridworld goal transfer and transition transfer gridworld alternating-T maze
Metrics: average episode score fraction of silent units cosine similarity between states receptive field shift population vector cosine similarity
Baselines: MF-only double DQN negative sampling only auxiliary loss
Builds on: double DQN contrastive predictive coding successor representation
Closest papers by meaning
- Predictive Coding Enhances Meta-RL To Achieve Interpretable Bayes-Optimal Belief Representation Under Partial Observability (NeurIPS 2025 · similarity 0.87)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
- A Multi-Region Brain Model to Elucidate the Role of Hippocampus in Spatially Embedded Decision-Making (ICML 2025 · similarity 0.85)
- Bridging State and History Representations: Understanding Self-Predictive RL (ICLR 2024 · similarity 0.84)
- Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning (NeurIPS 2025 · similarity 0.84)
- Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.