Research map / Reinforcement learning / Policy gradient and actor-critic
Learning mirror maps in policy mirror descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns ω-potential mirror maps for policy mirror descent via evolution strategies, showing learned maps beat negative entropy (NPG) across Grid-World, BCS, MinAtar and MuJoCo.
Problem
Policy mirror descent is usually instantiated with the negative-entropy mirror map (NPG), and existing theory suggests the mirror map has only mild influence on performance; it is unclear whether a different mirror map would perform better in practice.
Approach
Parameterize ω-potential mirror maps (a monotone neural parameterization for φ⁻¹ in tabular/MuJoCo settings, and a piecewise-linear φ for AMPO) and search over them with evolution strategies (OpenAI-ES, sep-CMA-ES), using the final policy's value as fitness.
Key results
- On Asterix and SpaceInvaders the learned mirror map's average return is more than double that of negative entropy (e.g., 170.24 vs 81.77 on SpaceInvaders when trained there).
- In Grid-World the learned mirror map ranked first in value across all tested configurations, ahead of the l2-norm and negative entropy, while having the worst theoretical error floor.
- The learned mirror map trained on Hopper significantly improves final performance over negative entropy in both train and test MuJoCo environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn the mirror map on Acrobot instead of using negative entropy | AMPO · Acrobot, CartPole, Asterix, Freeway, and SpaceInvaders | average return: -105.63 -> -88.49; 359.14 -> 476.41; 17.80 -> 24.03; 53.69 -> 56.00; 81.77 -> 144.01 | Helped |
| Learn the mirror map on CartPole instead of using negative entropy | AMPO · Acrobot, CartPole, Asterix, and SpaceInvaders | average return: -105.63 -> -83.76; 359.14 -> 499.93; 17.80 -> 27.26; 81.77 -> 100.07 | Helped |
| Learn the mirror map on CartPole instead of using negative entropy | AMPO · Freeway | average return: 53.69 -> 52.26 | Hurt |
| Learn the mirror map on Asterix instead of using negative entropy | AMPO · Acrobot, CartPole, Asterix, Freeway, and SpaceInvaders | average return: -105.63 -> -103.55; 359.14 -> 490.86; 17.80 -> 30.22; 53.69 -> 58.56; 81.77 -> 122.00 | Mixed |
| Learn the mirror map on Freeway instead of using negative entropy | AMPO · Acrobot, CartPole, and Freeway | average return: -105.63 -> -82.51; 359.14 -> 457.47; 53.69 -> 58.21 | Helped |
| Learn the mirror map on Freeway instead of using negative entropy | AMPO · Asterix and SpaceInvaders | average return: 17.80 -> 3.20; 81.77 -> 143.93 | Hurt |
| Learn the mirror map on SpaceInvaders instead of using negative entropy | AMPO · Acrobot, CartPole, and SpaceInvaders | average return: -105.63 -> -78.29; 359.14 -> 489.56; 81.77 -> 170.24 | Helped |
| Learn the mirror map on SpaceInvaders instead of using negative entropy | AMPO · Asterix and Freeway | average return: 17.80 -> 4.36; 53.69 -> 22.27 | Hurt |
| Use a learned mirror map rather than negative entropy | PMD in Grid-World · Grid-World | final value | Helped |
| Use the ℓ2-norm mirror map rather than negative entropy | PMD in Grid-World · Grid-World | final value | Helped |
| Use a learned mirror map rather than negative entropy | AMPO · Basic Control Suite and MinAtar | average return | Helped |
| Learn the mirror map on Hopper rather than use negative entropy | PMD · Hopper, Halfcheetah, and Ant | final performance | Helped |
Surprising findings
- The learned mirror map achieves the best value while having the worst theoretical error floor, suggesting PMD error-floor upper bounds are loose with respect to mirror map choice.
- The l2-norm mirror map consistently outperformed negative entropy in Grid-World.
Limitations
- Mirror maps are learned per environment, with ES fitness based on final return, which is computationally expensive.
- Theoretical guarantees do not yet reflect the impact of the mirror map on performance.
Experimental setup
Datasets: Grid-World Basic Control Suite MinAtar MuJoCo (brax Hopper, Halfcheetah, Ant)
Benchmarks: Grid-World Basic Control Suite MinAtar MuJoCo
Metrics: final policy value/return estimation error policy update distance
Baselines: negative entropy mirror map (NPG) l2-norm mirror map PMD with negative entropy AMPO with negative entropy
Builds on: policy mirror descent approximate mirror policy optimization natural policy gradient OpenAI-ES sep-CMA-ES
Closest papers by meaning
- Policy Mirror Descent with Lookahead (NeurIPS 2024 · similarity 0.92)
- Operator World Models for Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity (NeurIPS 2026 · similarity 0.89)
- Sharp Convergence and Sample Complexity of Policy Mirror Descent for Average-Reward MDPs (NeurIPS 2026 · similarity 0.88)
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.88)
- Convergence of Policy Mirror Descent Beyond Compatible Function Approximation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.