Research map / Reinforcement learning / Policy gradient and actor-critic

Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Off-policy actor-critic with expressive policies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Adaptive Horizon Actor-Critic (AHAC), a first-order model-based RL method on differentiable simulation that adapts rollout horizons to truncate at stiff contact, outperforming PPO/SAC/SVG/SHAC on locomotion tasks.

Problem

First-order model-based RL with differentiable simulation suffers from gradient sample error under stiff contact dynamics and long horizons, leading to suboptimal policies compared with model-free RL in contact-rich continuous control.

Approach

Adapt the model-based rollout horizon via a Lagrangian constraint so that rollouts are truncated when contact stiffness ||∇f|| exceeds a threshold, avoiding erroneous first-order gradients; combined with a double critic, iterative critic training, and GPU-parallel rollouts.

Key results

• AHAC reaches 40% higher PPO-normalized IQM reward than the best MFRL baseline PPO across five locomotion tasks (10 seeds).
• On SNU Humanoid (152 action dims), AHAC achieves 6746 raw reward vs. PPO 4114 and SHAC 5924.
• On Ant, AHAC achieves 9313 raw reward vs. PPO 6605 and SHAC 7662.

Measured effects reported in the paper

Surprising findings

• Model-free PPO and SAC do not reach AHAC's asymptotic reward even when given roughly 100x more samples and 10x more training time, suggesting zeroth-order gradient variance limits MFRL optima.
• Truncating SHAC's short horizon at stiff contact (graph truncation) did not improve performance in the parallel setting.

Limitations

• Requires a differentiable simulator (dflex), which limits applicability to real-world settings with non-differentiable or unavailable dynamics.
• Results rely on dflex, which is no longer actively developed; the paper leaves transfer to real robots as a challenge.
• The contact threshold C and iterative critic criteria were not tuned, and AHAC was under-tuned relative to SHAC.

Experimental setup

Datasets: Hopper Ant Anymal Humanoid SNU Humanoid

Benchmarks: Hopper Ant Anymal Humanoid SNU Humanoid

Metrics: 50% IQM episodic reward 95% confidence interval PPO-normalized reward score distributions

Baselines: PPO SAC SVG SHAC

Builds on: SHAC PPO SAC SVG

Closest papers by meaning

• Differentiable Information Enhanced Model-Based Reinforcement Learning (AAAI 2025 · similarity 0.88)
• Accelerated Policy Learning with Parallel Differentiable Simulation (ICLR 2022 · similarity 0.88)
• Gradient Information Matters in Policy Optimization by Back-propagating through Model (ICLR 2022 · similarity 0.88)
• Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control (ICLR 2024 · similarity 0.86)
• OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.85)