Research map / Multi-agent systems and games / Human-AI and goal-conditioned

ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making

NeurIPS 2026 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Embodied agent learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ManiTaskGen, a scene-graph-based generator of process- and outcome-based mobile manipulation tasks for arbitrary scenes, used to build benchmarks for VLM agents and a reflection-based inference-time improvement method.

Problem

Embodied agent training and evaluation rely on manually authored, scene-specific tasks, which limits coverage of the feasible task space across scenes.

Approach

Generate feasible mobile manipulation tasks automatically from a Receptacle-Aware 3D Scene Graph: process-based tasks by sampling and chaining atomic pick-and-place actions, and outcome-based tasks by instantiating 200 human-written templates filtered by a three-VLM vote. The same tasks are used to auto-build simulator benchmarks and to drive a self-reflection memory that improves VLM agents at in

Key results

• Human-verified validity: 94.0% process-based and 86.5% outcome-based tasks for ManiTaskGen-RAS-40K vs 29.4% and 21.1% for GPT-based generation.
• Best VLM (Gemini-2.5-pro) reaches 29.1% average SR vs 81.0% for humans on the benchmark.
• Self-reflection raises Gemini-2.5-flash SR from 36% to 58% on Level 1 tasks in a held-out scene.

Surprising findings

• Wrong placement is the largest source of Level 3 sub-step failures, suggesting spatial understanding is the main bottleneck.

Limitations

• Only navigate, pick, and place skills are covered; no fine-grained manipulation or non-pose state changes.
• Outcome-based (Level 4) tasks are not benchmarked due to subjective evaluation.
• Evaluation is limited to three scenes and a small number of trial episodes for improvement.

Experimental setup

Datasets: ReplicaCAD AI2THOR SUN-RGBD ManiTaskOT-200 Habitat ScanNet

Benchmarks: ManiTaskGen-RAS benchmark (Levels 1-3)

Metrics: success rate intermediate points human-verified validity rate

Baselines: GPTTaskGen ALFRED GenSim M3Bench EmbodiedBench Human

Builds on: Reflexion ReAct ALFRED

Closest papers by meaning

• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
• Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.89)
• PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.89)
• CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.89)
• Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.89)
• RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.88)