Research map / Reinforcement learning / Policy gradient and actor-critic

Target-Aligned Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Target-Aligned RL (TARL), which oversamples a minibatch and keeps the m transitions whose offline (target-network) and online TD errors agree most, improving DQN/DDQN/PER/ReaPER on MinAtar and Atari and SAC on MuJoCo-style tasks.

Problem

Target networks stabilize value-based deep RL but create a stability-recency tradeoff: slow target updates are stable but stale, hindering convergence.

Approach

A per-transition Target Alignment Score compares offline (target-network) and online TD errors; alignment-based oversampling draws m+b transitions and trains only on the m best-aligned ones, filtering stale or misaligned updates without changing the target or hyperparameters.

Key results

• TA-DDQN achieved a median peak score gain of 38.18% over DDQN on Atari-10 (mean nAUC gain 151.88%) with a 3.29% wall-clock overhead.
• TA-SAC improved mean nAUC by 10.65% across six continuous control environments, with 5 of 6 environments improved, at a 17.82% wall-clock increase.
• On MinAtar, TA-DQN improved nAUC over DQN in all four games, and TA-PER and TA-ReaPER also outperformed their non-aligned counterparts on average.

Surprising findings

• Doubling the batch size for vanilla DDQN underperformed both TA-DDQN and DDQN, indicating gains are not merely from extra forward passes.
• TA-SAC slightly underperformed SAC on Walker2d (-0.7 nAUC).

Limitations

• Naive TA-SAC adds about 17.8% wall-clock time because SAC does not natively perform online forward passes on sampled batches.
• Alignment can co-fail since the online and target networks share an ancestor; alignment is necessary but not sufficient.
• Alternatives such as prioritized sampling and loss weighting were not evaluated; only oversampling was studied.

Experimental setup

Datasets: Atari-10 MinAtar Ant-v4 BipedalWalker-v3 Hopper-v4 Humanoid-v4 Swimmer-v4 Walker2d-v4

Benchmarks: Atari-10 MinAtar Ant-v4 BipedalWalker-v3 Hopper-v4 Humanoid-v4 Swimmer-v4 Walker2d-v4

Metrics: normalized area under the curve (nAUC) peak score gain wall-clock time overhead batch alignment

Baselines: DDQN DQN SAC PER ReaPER DQN (2m) doubled batch size

Builds on: DQN Double DQN Prioritized Experience Replay ReaPER Soft Actor-Critic

Closest papers by meaning

• Learning the Target Network in Function Space (ICML 2024 · similarity 0.87)
• Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.85)
• ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.85)
• Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.84)
• STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning (NeurIPS 2025 · similarity 0.84)