Research map / Reinforcement learning / Visual and representation RL
AssistanceZero: Scalably Solving Assistance Games
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AssistanceZero, an AlphaZero extension with learned reward-parameter and human-action prediction heads, to solve assistance games in a new Minecraft building benchmark (MBAG), outperforming PPO, pretraining and SFT baselines and reducing human actions in a 16-person study.
Problem
Assistance games are hard to scale: the assistant must plan under uncertainty over a huge reward-parameter space, and it needs an accurate human model. Prior work used only toy settings with few goals.
Approach
Extend AlphaZero with a recurrent network that has policy, value, reward-parameter prediction and human-action prediction heads, so MCTS can plan under uncertainty by simulating human actions and rewards from learned beliefs, separating goal prediction from action selection.
Key results
- AssistanceZero assistant raises overall goal completion to 79.8% vs 71.6% for the PPO baseline and cuts human actions from 203 to 158 in the imitation-learning human model setting.
- Assistance-game assistant builds about 26% of the goal itself and reduces human actions to 179 vs 240-241 for pretraining and SFT baselines.
- In a 16-subject human study, AssistanceZero helpfulness was rated 3.1/5 vs 1.7 for SFT and 4.0 for the expert human assistant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AssistanceZero instead of PPO | Assistant in MBAG · MBAG test set | Assistant goal %: 0.0 ± 0.8 -> 27.0 ± 1.5 | Helped |
| Use AssistanceZero instead of PPO | Assistant in MBAG · MBAG test set | Human actions: 203 ± 3 -> 158 ± 3 | Helped |
| Remove test-time MCTS (ablation) | AssistanceZero · MBAG test set | Assistant goal %: 27.0 ± 1.5 -> 27.3 ± 1.3 | No clear effect |
| Replace pretraining or SFT with assistance-game training | MBAG assistant · MBAG test set | Assistant goal %: 2.3 ± 0.5; 2.9 ± 0.3 -> 26.0 ± 3.3 | Helped |
| Replace pretraining or SFT with assistance-game training | MBAG assistant · MBAG test set | Human actions: 240 ± 4; 241 ± 4 -> 179 ± 11 | Helped |
| Remove LSTM (ablation) | AssistanceZero · MBAG test set | Overall goal %: 77.5 ± 3.2 -> 69.0 ± 3.6 | Hurt |
| Remove the previous-reward KL penalty (ablation) | AssistanceZero · MBAG test set | Assistant goal %: 25.2 ± 4.6 -> 18.1 ± 5.1 | Hurt |
| Remove LSTM (ablation) | BC-combined human model · Human action prediction | Cross entropy (alone): 1.89 -> 2.13 | Hurt |
| Remove data augmentation (ablation) | BC-combined human model · Human action prediction | Cross entropy (alone): 1.89 -> 2.41 | Hurt |
| Remove dropout (ablation) | BC-combined human model · Human action prediction | Cross entropy (alone): 1.89 -> 2.56 | Hurt |
| Set piKL c_PUCT to 10 instead of 30 (ablation) | piKL-combined human model · Human action prediction | Cross entropy (alone): 1.98 -> 2.28 | Hurt |
| Set piKL c_PUCT to 50 instead of 30 (ablation) | piKL-combined human model · Human action prediction | Cross entropy (alone): 1.98 -> 1.91 | No clear effect |
Surprising findings
- PPO assistants trained with the shared reward failed to help at all (assistant goal ~0%), converging to building little or nothing.
- Reward-based human models (PPO, AlphaZero) were poor predictors of real human actions and built houses faster than human subjects.
Limitations
- Evaluated only in a single Minecraft building environment with simplified house goals.
- Large gap remains between the AssistanceZero assistant and the expert human assistant.
- Human model is a fixed piKL/BC model and the paper does not compare to full RLHF because it is hard to apply in MBAG.
Experimental setup
Datasets: CraftAssist houses dataset human subject episodes collected for MBAG
Benchmarks: MBAG
Metrics: overall goal percentage human actions assistant goal percentage helpfulness Likert rating cross entropy goal percentage after X minutes
Baselines: PPO pretraining SFT PPO with reward engineering BC human models piKL human models
Builds on: AlphaZero piKL PPO MuZero POMCP
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.87)
- Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning (CVPR 2026 · similarity 0.86)
- Learning to Assist Humans without Inferring Rewards (NeurIPS 2024 · similarity 0.85)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.85)
- Experience-based Knowledge Correction for Robust Planning in Minecraft (ICLR 2026 · similarity 0.85)
- GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.