Research map / Reinforcement learning / Policy gradient and actor-critic
Operator World Models for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POWR, which learns a conditional-mean-embedding world model of the MDP and uses it in closed form to estimate action-value functions for policy mirror descent, with convergence guarantees and small Gym experiments.
Problem
Policy mirror descent needs exact action-value functions for every iterate, which are inaccessible in RL; repeated sampling to estimate them is expensive and requires unrealistic simulator access.
Approach
Learn the transition operator and reward via conditional mean embeddings (CMEs) and derive a closed-form kernel-based estimator of q_pi for any compatible policy via matrix operations, so no resampling is needed per PMD iterate; combine with KL-based PMD (POWR).
Key results
- POWR outperforms A2C, DQN, TRPO and PPO in sample complexity on Gym FrozenLake-v1, Taxi-v3 and MountainCar-v0 across 7 runs.
- On Taxi-v3, POWR avoids the local optimum that other methods (except DQN) converge to.
- Theory: inexact PMD with decreasing error O(1/t) gives O(log T / T) rate; main theorem gives O(1/T + delta^2 n^{-alpha}) suboptimality with high probability.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use POWR instead of the baselines | POWR · FrozenLake-v1 | cumulative reward | Helped |
| Use POWR instead of the baselines | POWR · Taxi-v3 | cumulative reward | Helped |
| Use POWR instead of the baselines | POWR · MountainCar-v0 | cumulative reward | Helped |
| Use POWR instead of A2C, TRPO, and PPO | POWR · Taxi-v3 | convergence to a local optimum | Helped |
| Use POWR instead of other approaches | POWR · Test environments | training stability | Hurt |
Surprising findings
- The method is less stable than baselines, particularly in early training, which the authors attribute to exploration-exploitation interplay.
Limitations
- Experiments are small-scale Gym environments only; scalability requires Nystrom or reduced-rank approximations not evaluated.
- Action space must be finite; infinite action PMD left open.
- Training is less stable than baselines early on; exploration-exploitation alternation not investigated theoretically.
Experimental setup
Datasets: FrozenLake-v1 Taxi-v3 MountainCar-v0
Benchmarks: FrozenLake-v1 Taxi-v3 MountainCar-v0
Metrics: average cumulative reward timesteps to success threshold
Baselines: A2C DQN TRPO PPO
Builds on: Policy Mirror Descent Conditional Mean Embeddings Inexact PMD (Lan 2021 / Xiao 2022 style analysis) Simulation lemma
Closest papers by meaning
- Learning mirror maps in policy mirror descent (ICLR 2025 · similarity 0.91)
- Policy Mirror Descent with Lookahead (NeurIPS 2024 · similarity 0.90)
- Sharp Convergence and Sample Complexity of Policy Mirror Descent for Average-Reward MDPs (NeurIPS 2026 · similarity 0.88)
- Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity (NeurIPS 2026 · similarity 0.87)
- Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence (NeurIPS 2026 · similarity 0.87)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.