Research map / Reinforcement learning / Policy gradient and actor-critic

Actor-Free Continuous Control via Structurally Maximizable Q-Functions

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Off-policy actor-critic with expressive policies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q3C, an actor-free value-based continuous control method that learns a wire-fitting control-point Q-function with architectural and training fixes, matching TD3 on standard MuJoCo tasks and beating it on action-restricted environments.

Problem

Off-policy value-based RL struggles with continuous action maximization; actor-critic methods (TD3, SAC) suffer instability and local optima when Q-functions are non-convex, such as in action-constrained environments.

Approach

Revisits wire-fitting control-point interpolation as a deep Q-function that is structurally maximized at one of its learned control-points, adding action-conditioned Q-value generation, top-k relevance filtering, a pairwise separation loss for control-point diversity, and normalized/annealed smoothing, with TD3-style exploration and twin critics.</303>

Key results

• On restricted environments Q3C reaches 1000±0 on InvertedPendulumBox, 4357.82 on HalfCheetahBox and 1974.28 on HopperBox, beating TD3 (782.76, 2276.70, 1406.83).
• On standard MuJoCo tasks Q3C is close to TD3, e.g. Hopper 3206.14 vs 3113.41 and Walker2d 3977.39 vs 4770.82, and clearly above NAF and RBF-DQN on most tasks.
• Ant-v4 is a weak point: Q3C reaches 3698.41 versus TD3's 5167.68.

Measured effects reported in the paper

Surprising findings

• Q3C's wall-clock time advantage is largely offset by the interpolation overhead, so removing the actor does not reduce runtime in the reported experiments.
• NAF slightly beats Q3C on HalfCheetahBox, suggesting a quadratic approximation may suffice there.

Limitations

• Sample efficiency can lag behind other baselines in some environments, and Ant-v4 performance is below TD3.
• Exploration simply reuses TD3's Gaussian noise rather than a dedicated exploration scheme.
• Evaluated only on deterministic Q-learning; no stochastic/soft variant or offline setting.

Experimental setup

Datasets: Gymnasium MuJoCo tasks Restricted hypersphere-action variants (InvertedPendulumBox, HopperBox, HalfCheetahBox) Adroit Hand Hammer

Benchmarks: Pendulum-v1 Swimmer-v4 Hopper-v4 BipedalWalker-v3 Walker2d-v4 HalfCheetah-v4 Ant-v4 InvertedPendulumBox HalfCheetahBox HopperBox

Metrics: average episodic return wall-clock time PyTorch memory allocation

Baselines: TD3 NAF Wire-Fitting RBF-DQN SAC PPO

Builds on: TD3 wire-fitting RBF-DQN NAF

Closest papers by meaning

• Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.89)
• Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.89)
• Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic (ICML 2024 · similarity 0.89)
• Stochastic Q-learning for Large Discrete Action Spaces (ICML 2024 · similarity 0.88)
• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
• Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.88)