Research map / Large language models / Agents, code and math
CraftFactory: A Conditioned Control Policy Benchmark for Compositional Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CraftFactory, a Minecraft crafting-GUI benchmark for compositional generalization in interactive sequential control, and evaluates Meta-Learning (MLC), Self Attention (EFMDP), and Equivariant Map (HOMW) methods on a VPT-based FiLM-conditioned policy.
Problem
Existing compositional generalization benchmarks are mostly symbolic language or grid-world translation tasks without interactive control, so they do not test whether agents generalize compositional instructions through embodied action in a visual environment.
Approach
Frames compositional generalization as instruction-conditioned sequential control in a Minecraft crafting GUI, where agents must execute unseen combinations of item-slot drag-and-drop actions from pixel observations, with a VPT backbone and FiLM conditioning used as a baseline.
Key results
- Equivariant Map reached 31.3 ± 3.2% on Task-1 and 19.3 ± 5.3% on Task-2, while CraftFactory (the authors' VPT+FiLM baseline) reached 35.7 ± 6.7% on Task-1 but only 4.5% on Task-2.
- On the extrapolated multi-step tasks, all methods collapse: Task-5 success rates are 0.0-0.9%.
- The authors conclude existing compositional generalization methods, including their own, do not achieve high performance on reasoning-to-practice control tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CraftFactory instead of Equivariant Map | CraftFactory control benchmark · CraftFactory Task-1 (in-distribution) | success rate: 31.3 ± 3.2 -> 35.7± 6.7 | Helped |
| Use CraftFactory instead of Meta-Learning | CraftFactory control benchmark · CraftFactory Task-2 (extrapolation) | success rate: 25.5 ± 3.0 -> 4.5 | Hurt |
| Use CraftFactory instead of Self Attention | CraftFactory control benchmark · CraftFactory Task-3 (in-distribution) | success rate: 11.5 ± 6.8 -> 7.4 ± 2.6 | Hurt |
| Use CraftFactory instead of Equivariant Map | CraftFactory control benchmark · CraftFactory Task-4 (extrapolation) | success rate: 5.2 ± 3.9 -> 2.7 | Hurt |
| Use CraftFactory instead of Meta-Learning | CraftFactory control benchmark · CraftFactory Task-5 (extrapolation) | success rate: 0.9 -> 0.9 | No clear effect |
| Generalize from one-step tasks to multi-step tasks | CraftFactory methods · CraftFactory | success rate | Hurt |
Surprising findings
- Re-implemented methods that perform well on symbolic benchmarks fall to near-zero success on multi-step control generalization (Task-5 at most 0.9%).
Limitations
- Baselines were re-implemented with an adaptive ResConnect head, which may not reflect their peak performance.
- Recognition errors dominate failures in the basic tasks.
- Long-horizon tasks suffer from an amnesia problem.
Experimental setup
Datasets: CraftFactory random drag-and-drop trajectories (6,000) CraftFactory item-specific drag trajectories (4,000)
Benchmarks: CraftFactory Task-1 to Task-5
Metrics: success rate
Baselines: MLC meta-learning EFMDP self-attention HOMW equivariant map
Builds on: VPT FiLM MineRL MLC HOMW
Closest papers by meaning
- Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning (ICLR 2026 · similarity 0.88)
- Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.88)
- Attention as a Hypernetwork (ICLR 2025 · similarity 0.88)
- Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Scaling can lead to compositional generalization (NeurIPS 2025 · similarity 0.87)
- When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.