Research map / Reinforcement learning / Policy gradient and actor-critic
Value Iteration in Continuous Actions, States and Time
Michael Lutter, Shie Mannor, Jan Peters, Dieter Fox, Animesh Garg
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use offline cFVI instead of the strongest listed deep RL baseline | Swing-up control · Simulated Pendulum | reward: -030.6 ± 001.4 -> -030.5 ± 000.8 | Improved |
| Use offline cFVI instead of the strongest listed deep RL baseline | Swing-up control · Simulated Cartpole | reward: -024.2 ± 001.4 -> -024.2 ± 002.1 | No clear effect |
| Use offline cFVI instead of the strongest listed deep RL baseline | Swing-up control · Simulated Furuta Pendulum | reward: -028.1 ± 002.0 -> -027.7 ± 001.6 | Improved |
| Use offline cFVI instead of PPO with uniform initial states and domain randomization | Swing-up control · Physical Cartpole | reward: -143.8 ± 016.1 -> -143.7 ± 210.4 | No clear effect |
| Use offline cFVI instead of PPO with uniform initial states and domain randomization | Swing-up control · Physical Furuta Pendulum | reward: -080.6 ± 010.8 -> -082.1 ± 007.6 | No clear effect |
| Use RTDP cFVI instead of offline DP cFVI | cFVI · Physical Furuta Pendulum | reward: -082.1 ± 007.6 -> -1009.9 ± 004.5 | Hurt |
| Increase the n-step horizon using larger λ (ablation) | DP cFVI · Swing-up control tasks | convergence rate | Improved |
| Use an extremely long value-target horizon (ablation) | DP cFVI · Furuta Pendulum and Cartpole | learning stability | Hurt |
| Increase the value-target horizon from λ = 0.01 to λ = 0.99 (ablation) | cFVI · Pendulum | forward rollout time: 0.4s -> 56.4s | Hurt |
| Use a locally quadratic value function rather than an MLP (ablation) | cFVI · Control tasks | learning speed | Improved |
| Use a standard network architecture without the locally quadratic structure or feature transform (ablation) | cFVI · Cartpole | policy quality | Hurt |
| Add the feature transform to the locally quadratic value function (ablation) | cFVI · Cartpole | policy quality | Improved |
Closest papers by meaning
- Continuous-Time Fitted Value Iteration for Robust Policies (TPAMI 2022 · similarity 0.92)
- Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control (NeurIPS 2025 · similarity 0.87)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.87)
- Sketched Newton Value Iteration for Large-Scale Markov Decision Processes (AAAI 2024 · similarity 0.86)
- A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
