Research map / Reinforcement learning / Imitation learning

Imperfect World Models are Exploitable

NeurIPS 2026 · Reinforcement learning › Imitation learning · approach: Reward learning from demonstrations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines model exploitation as a value inversion between an imperfect and true transition model, proves it is unavoidable on policy sets with open subsets, shows unexploitability conditions fail on finite policy sets, and derives a safe horizon for a relaxed epsilon-exploitation notion.

Problem

Whether imperfect world models (transition models) can be guaranteed to preserve the ordering of policies induced by the true model, i.e., whether model exploitation can be avoided, given that reward hacking results do not transfer directly.

Approach

Defines exploitability as an ordinal value inversion between two transition models; shows via rationality of the value function in policy space that any non-trivial, non-equivalent pair is exploitable on policy sets containing an open subset, reducing exploitation to reward hacking; introduces epsilon-exploitation and a safe horizon from a tight simulation lemma.

Key results

• Every non-trivial, non-equivalent pair of transition models is exploitable on any policy set containing an open subset (including all stationary policies, epsilon-suboptimal policies, and delta-deterministic policies).
• Non-trivial, non-equivalent unexploitable pairs cannot be guaranteed on finite policy sets; a counterexample shows all pairs can be equivalent on a three-policy set.
• The safe horizon H(eps, delta) gives epsilon-unexploitability whenever 1/(1-gamma) <= H, and this bound is tight.

Surprising findings

• Exploitation is inevitable on any policy set with an open subset, unlike the finite-policy case for reward hacking.
• Hacking does not imply exploitation: a counterexample shows hackable reward pairs whose ordering cannot be altered by any transition perturbation.

Limitations

• Restricted to finite state and action MDPs.
• Binary exploitation notion is restrictive; relaxed epsilon-exploitation only partially addresses this.
• No closed-form necessary and sufficient conditions for unexploitability on finite policy sets.

Experimental setup

Builds on: reward hacking theory (Skalse et al.) simulation lemma value equivalence principle

Closest papers by meaning

• Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.86)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.86)
• Reward-Free Curricula for Training Robust World Models (ICLR 2024 · similarity 0.86)
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.86)
• Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.85)
• Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.84)