Research map / Reinforcement learning / Imitation learning

Inverting the Bellman Equation: From $Q$-Values to World Models

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Reward learning from demonstrations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that value-based agents trained on rich goal sets implicitly encode a unique world model, and introduces P-learning to extract it by inverting the Bellman equation, validated on FourRooms, Reacher and MountainCar.

Problem

Model-free value-based agents are thought not to encode the transition dynamics uniquely because value equivalence allows many world models to yield identical Q-values; the question is when goal-conditioned agents implicitly encode an accurate model of the environment.

Approach

P-learning is an inverse of Q-learning that fits a candidate transition kernel to fixed Q-values, policies and known rewards by minimising the Bellman residual over the model. Theory gives goal-set conditions under which this kernel is uniquely identifiable (deterministic/stochastic, finite/continuous), so a sufficiently rich set of goals makes the Q-function collection informationally equivalent

Key results

• On Reacher with |G|=4 goals, the extracted world model reaches NMSE 1.2e-4 despite Q-values with NMSE 5.7e-1.
• Spearman correlation between agent return and world model error on Reacher is rho = -0.98 across architecture capacities.
• A Reacher world model trained on position goals yields policies that are quasi-optimal on out-of-distribution velocity goals.

Surprising findings

• Policies trained exclusively inside a position-trained Reacher world model are quasi-optimal on velocity-based goals.
• Velocity-goal and position-goal MountainCar agents encode near-identical world models.

Limitations

• Theory for stochastic continuous MDPs holds only for unconditional policies.
• Does not cover partially observable MDPs or agents without Q-values.
• Scalability to very large finite environments not tested.

Experimental setup

Datasets: FourRooms Reacher MountainCar

Benchmarks: FourRooms Reacher MountainCar

Metrics: NMSE MSE mean return Spearman correlation

Baselines: successor features Q-learning

Builds on: Q-learning PQN Hindsight Experience Replay value equivalence

Closest papers by meaning

• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
• Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.89)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.88)
• Calibrated Value-Aware Model Learning with Probabilistic Environment Models (ICML 2025 · similarity 0.88)
• Reward-Free Curricula for Training Robust World Models (ICLR 2024 · similarity 0.88)