Research map / Reinforcement learning / Policy gradient and actor-critic
How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train critic on action-gradient of TD error (MAGE) instead of plain TD error (Dyna) | TD3 with learned model · HalfCheetah-v2 asymptotic | maximum average return: 8372.76 -> 9660.79 | Helped |
| add action-gradient loss to critic training (MAGE vs Dyna) | TD3 critic with oracle dynamics · Pendulum-v0 action-gradient error | action-gradient error | Helped |
| set lambda to 0 (drop TD-error regularization) (ablation) | MAGE-DDPG · Pendulum-v0 | critic learning | Hurt |
| use learned reward function instead of ground-truth reward (ablation) | MAGE-TD3 · MuJoCo tasks (Pusher-v2) | average return | No clear effect |
| reduce model capacity (2-member ensemble, 4x256) (ablation) | MAGE-TD3 · Pusher-v2 | average return | No clear effect |
| reduce model to single model 2x256 (ablation) | MAGE-TD3 · Pusher-v2 | average return | Hurt |
| vary lambda in [0,1] (ablation) | MAGE-TD3 · HalfCheetah-v2 | median return | No clear effect |
| increase updates per step to 10 for model-free TD3 | TD3 · Pusher-v2 | average return | Hurt |
Closest papers by meaning
- Wasserstein Policy Optimization (ICML 2025 · similarity 0.89)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.89)
- Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning (ICML 2024 · similarity 0.88)
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.88)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.88)
- ACE: Off-Policy Actor-Critic with Causality-Aware Entropy Regularization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.