Research map / Reinforcement learning / Policy gradient and actor-critic
Bridging the performance-gap between target-free and target-based reinforcement learning
Théo Vincent, Yogesh Tripathi, Tim Faust, Abdullah Akgül, Yaniv Oren, Melih Kandemir, Jan Peters, Carlo D'Eramo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| share features and keep only a copy of the last linear head as target (iS-DQN K=1) | target-free DQN, CNN + LayerNorm, no normalization · Atari 15 games | AUC gap vs target-based: 60% gap -> 18% gap | Improved |
| learn more Bellman iterations in parallel (K=9) with shared heads (ablation) | iS-DQN, CNN + LayerNorm · Atari 15 games | AUC vs target-based: 10% below -> 6% above target-based | Improved |
| increase heads to K=49 (ablation) | iS-DQN, CNN + LayerNorm · Atari 15 games | AUC | Hurt |
| increase heads to K=49 with the richer IMPALA architecture (ablation) | iS-DQN, IMPALA + LayerNorm · Atari 10 games | AUC | Improved |
| remove LayerNorm and raise K above 3 (ablation) | iS-DQN, CNN · Atari 15 games | AUC: K=3 small benefit, slightly decreasing for higher K | Hurt |
| add iterated shared heads to CQL (iS-CQL K=9) | offline CQL · Atari 10 games IMPALA, 10% data | AUC gap vs target-based: 26% -> 6% | Improved |
| use ensemble of target/online head pairs (ES-CQL) instead of iterated heads | offline CQL · Atari 10 games IMPALA, 10% data | AUC | Improved |
| use old copy of last critic layer as target (iS-SAC K=1) | target-free SAC with SimbaV2 · DMC Hard 7 tasks | performance: recovers performance drop; 49% fewer total parameters | Improved |
| increase heads from K=1 to K=9 for SAC (ablation) | iS-SAC · DMC Hard 7 tasks | performance | No clear effect |
| add iS-QN K=9 to ILQL | target-free ILQL · Wordle | learning speed: +10% | Improved |
| increase heads from K=1 to K=3 in streaming Q(lambda) | Stream Q(lambda) · Atari 7 games streaming | learning speed: more than 10% | Improved |
| increase heads to K=9 in streaming (ablation) | iS-Stream Q(lambda) · Atari 7 games streaming | learning speed | No clear effect |
Closest papers by meaning
- Target-Aligned Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Learning the Target Network in Function Space (ICML 2024 · similarity 0.87)
- From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.86)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
