Research map / Reinforcement learning / Policy gradient and actor-critic
Self-Consistent Models and Values
Gregory Farquhar, Kate Baumli, Zita Marinho, Angelos Filos, Matteo Hessel, Hado van Hasselt, David Silver
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SC-Direct rather than using only grounded updates | Muesli · Atari games and Sokoban | sample efficiency | Improved |
| use SC-Direct instead of Dyna-style value-only planning | Muesli with a value-equivalent model · Atari games and Sokoban | performance | Improved |
| use a residual self-consistency objective instead of a semi-gradient objective (ablation) | Muesli · Atari games and Sokoban | performance | Hurt |
| use SC-Reverse instead of SC-Direct (ablation) | Muesli · Atari games and Sokoban | sample efficiency | Mixed |
| use SC-Residual or SC-Reverse instead of SC-Direct (ablation) | tabular Garnet MDP policy evaluation · Garnet MDP | value-learning speed | Mixed |
| use SC-Direct rather than the other tested updates (ablation) | tabular Garnet MDP control · Garnet MDP | final policy value | Improved |
| replay the environment's original action sequence instead of resampling actions from the policy (ablation) | SC-Direct · Atari games | performance | Hurt |
| add epsilon exploration to the imagined policy with epsilon = 0.5 (ablation) | SC-Direct · Assault and Asteroids | performance | Improved |
| add epsilon exploration to the imagined policy with epsilon = 0.5 (ablation) | SC-Direct · Most tested environments | performance | No clear effect |
| perturb starting latent states with multiplicative noise (ablation) | SC-Direct · Atari games | performance | No clear effect |
| combine latent-state perturbations with epsilon = 0.5 exploration (ablation) | SC-Direct · Atari games | performance | Improved |
| add self-consistency as an auxiliary task (ablation) | policy-gradient agent with a value-equivalent auxiliary model · Atari representation learning | episode return | Improved |
Closest papers by meaning
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.87)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.87)
- Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach (NeurIPS 2025 · similarity 0.87)
- Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning (ICLR 2025 · similarity 0.86)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
