Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning to Build by Building Your Own Instructions
Aaron T Walsman, Muru Zhang, Adam Fishman, Ali Farhadi, Dieter Fox
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use instruction stack explicit memory instead of LSTM or causal transformer history | InstructioNet · Break and Make RC-8 | F1_a: 0.12 -> 0.63 vs Studnet-B | Improved |
| train with online imitation (alpha=0.75) instead of behavior cloning (alpha=1.0) (ablation) | InstructioNet · Break and Make RC-4 | F1_a: 0.66 -> 0.71 | Improved |
| use summed-probability cross entropy cursor loss instead of BCE or MSE (ablation) | InstructioNet · Break and Make RC-2 | F1_a: 0.72 (BCE), 0.66 (MSE) -> 0.93 | Improved |
| condition click heads on sampled action and parameters (ablation) | InstructioNet · Break and Make RC-2 | F1_e: 0.00 -> 0.95 (at 2.6M frames) | Improved |
| use expert-generated instruction images instead of model-generated (ablation) | InstructioNet · Break and Make RC-Vehicles | F1_a: 0.53 -> 0.65 | Improved |
| add camera motion between current and instruction images (ablation) | InstructioNet · Break and Make RC-Vehicles | F1_a: 0.53 -> 0.33 (trained), 0.00 (untrained) | Hurt |
| ask the agent to rebuild with an altered brick color | InstructioNet · Break and Make RC-4 | F1_a: 0.71 -> 0.68 | Hurt |
Closest papers by meaning
- Break and Make: Interactive Structural Understanding Using LEGO Bricks (ECCV 2022 · similarity 0.89)
- Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning (ICML 2022 · similarity 0.88)
- Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning (NeurIPS 2021 · similarity 0.87)
- BrickNet: Graph-Backed Generative Brick Assembly (CVPR 2026 · similarity 0.86)
- Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks (NeurIPS 2024 · similarity 0.84)
- Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
