AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Bridging the performance-gap between target-free and target-based reinforcement learning

Théo Vincent, Yogesh Tripathi, Tim Faust, Abdullah Akgül, Yaniv Oren, Melih Kandemir, Jan Peters, Carlo D'Eramo

ICLR 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
share features and keep only a copy of the last linear head as target (iS-DQN K=1)target-free DQN, CNN + LayerNorm, no normalization · Atari 15 gamesAUC gap vs target-based: 60% gap -> 18% gapImproved
learn more Bellman iterations in parallel (K=9) with shared heads (ablation)iS-DQN, CNN + LayerNorm · Atari 15 gamesAUC vs target-based: 10% below -> 6% above target-basedImproved
increase heads to K=49 (ablation)iS-DQN, CNN + LayerNorm · Atari 15 gamesAUCHurt
increase heads to K=49 with the richer IMPALA architecture (ablation)iS-DQN, IMPALA + LayerNorm · Atari 10 gamesAUCImproved
remove LayerNorm and raise K above 3 (ablation)iS-DQN, CNN · Atari 15 gamesAUC: K=3 small benefit, slightly decreasing for higher KHurt
add iterated shared heads to CQL (iS-CQL K=9)offline CQL · Atari 10 games IMPALA, 10% dataAUC gap vs target-based: 26% -> 6%Improved
use ensemble of target/online head pairs (ES-CQL) instead of iterated headsoffline CQL · Atari 10 games IMPALA, 10% dataAUCImproved
use old copy of last critic layer as target (iS-SAC K=1)target-free SAC with SimbaV2 · DMC Hard 7 tasksperformance: recovers performance drop; 49% fewer total parametersImproved
increase heads from K=1 to K=9 for SAC (ablation)iS-SAC · DMC Hard 7 tasksperformanceNo clear effect
add iS-QN K=9 to ILQLtarget-free ILQL · Wordlelearning speed: +10%Improved
increase heads from K=1 to K=3 in streaming Q(lambda)Stream Q(lambda) · Atari 7 games streaminglearning speed: more than 10%Improved
increase heads to K=9 in streaming (ablation)iS-Stream Q(lambda) · Atari 7 games streaminglearning speedNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy