Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Position: Scaling Simulation is Neither Necessary Nor Sufficient for In-the-Wild Robot Manipulation

ICML 2024 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Embodied agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A position paper arguing that scaling robotic physics simulators is neither necessary nor sufficient for zero-shot, human-preference-compliant in-the-wild robot manipulation, and advocating passive web video, safety constraints, and real-world data collection instead.

Problem

Whether scaling physics simulation is a productive route to general-purpose, zero-shot, human-preference-compliant real-world robot manipulation.

Approach

Argues via conceptual arguments and cited prior evidence that simulators cannot match real-world physics and contact diversity, that simulation-based progress can be a false signal for real-world manipulation, and that passive human video, task-agnostic safety constraints, and scalable real-world data collection are more promising than scaling simulation.

Key results

• Cites Dasari et al. (2023) finding that visual representations performing well in simulated manipulation do not transfer well to the real world, with weak correlation between simulation and real performance.
• Argues that simulating pouring water or chopping is harder than executing the task, so simulation is not necessary for such everyday manipulation.

Surprising findings

• Simulating everyday tasks such as pouring water can be harder than executing them, so simulation may solve a harder problem than needed.

Limitations

• Argument is conceptual and relies largely on cited prior work rather than new experiments
• Scope restricted to in-the-wild manipulation; does not apply to structured industrial settings
• Acknowledges simulation can be useful for prototyping RL algorithms

Experimental setup

Builds on: domain randomization RLHF

Closest papers by meaning

• Rapidly Adapting Policies to the Real-World via Simulation-Guided Fine-Tuning (ICLR 2025 · similarity 0.87)
• RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills (NeurIPS 2025 · similarity 0.86)
• GENMANIP: LLM-driven Simulation for Generalizable Instruction-Following Manipulation (CVPR 2025 · similarity 0.86)
• ASID: Active Exploration for System Identification in Robotic Manipulation (ICLR 2024 · similarity 0.85)
• DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control (ICML 2025 · similarity 0.85)
• Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.85)