Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Discovery of Useful Questions as Auxiliary Tasks
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, Satinder Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hand-crafted auxiliary tasks (reward prediction, pixel control, random GVFs) with meta-learned GVF questions, representation trained by auxiliary only | IMPALA / A2C agent · Collect-Objects, Puddleworld, Atari | mean episode return | Improved |
| use multi-step (non-myopic) meta-gradient with k>2 instead of myopic k=1 (ablation) | Discovered GVFs · Collect-Objects | episode return | Improved |
| increase unroll length from k=25 to k=50 (ablation) | Discovered GVFs · Collect-Objects | final and average return | Hurt |
| use too few questions (d=2) (ablation) | Discovered GVFs · Collect-Objects | final return | Hurt |
| use too many questions (d=128 vs smaller) (ablation) | Discovered GVFs · Collect-Objects | average return | Hurt |
| meta-loss as area under curve (summed) rather than end-batch only (ablation) | Discovered GVFs · Atari (3 games) | episode return stability | Improved |
| add discovered GVF auxiliary tasks jointly to IMPALA | IMPALA · Atari top-10 pixel-control games | relative performance gain: large gains in 6 of 10, small gains in 2, losses in 2 | Mixed |
| add reward prediction auxiliary task to IMPALA | IMPALA · Atari 57 | median normalized score | No clear effect |
Closest papers by meaning
- EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL (NeurIPS 2022 · similarity 0.87)
- HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation (ICLR 2026 · similarity 0.86)
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.85)
- AGILE: A Novel Reinforcement Learning Framework of LLM Agents (NeurIPS 2024 · similarity 0.85)
- Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.84)
- ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards (ICML 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
