Research map / Multi-agent systems and games / LLM agents

TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TaskGround, a training-free Ground-Infer-Execute pipeline that grounds full household scenes into compact slices, infers and completes executable goal sequences, and compiles them into skill actions, evaluated on the 400-task FullHome benchmark.

Problem

Household agents must infer task-relevant entities, implicit goals, and process constraints from a complete household scene and a situated request, but direct complete-scene prompting is costly and error-prone, especially for compact open-weight models.

Approach

A training-free three-stage decomposition: LLM-based node selection over an entity inventory with deterministic scene-slice reconstruction, LLM goal-sequence inference plus rule-based household-prior completion, and a fixed skill-level executor that realizes goals as actions.

Key results

• On FullHome VirtualHome, Qwen3.5-9B Goal SR rises from 0.0 (Naive) to 47.5 with TaskGround, comparable to GPT-5 Naive at 45.5.
• GPT-5 Goal SR on VirtualHome improves from 45.5 to 73.5 with TaskGround.
• TaskGround reduces total input tokens by about 17.9x-18.0x on VirtualHome versus full-scene prompting (5.8x and 3.9x on BEHAVIOR subsets).

Surprising findings

• Ground+Act yields higher recall than Retrieve+Act but lower task recovery for Qwen3.5-9B on process tasks, showing grounding recall alone is insufficient.
• Naive prompting often has high precision but low recall, especially for Qwen3.5-9B.

Limitations

• FullHome is a controlled simulator setting with structured scenes and predefined skills, without perception noise or partial observability.
• Completion relies on fixed hand-specified household priors, and the GT-scene-slice diagnostic shows executable task-structure inference remains challenging.
• VirtualHome split uses a single underlying home with per-task perturbations.

Experimental setup

Datasets: VirtualHome BEHAVIOR

Benchmarks: FullHome

Metrics: Goal SR Process SR Task Recall Scene entity recall/precision input token cost

Baselines: Naive complete-scene prompting Ground-Act Retrieve+Act Ground+Act

Builds on: VirtualHome BEHAVIOR

Closest papers by meaning

• PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.88)
• OntoPlan: An Ontology-Grounded Scene Representation and Agentic Framework for Scalable Robot Task Planning (NeurIPS 2026 · similarity 0.86)
• HomeFlow: A Data Flywheel for Smart Home Agent Training with Verifiable Simulation (NeurIPS 2026 · similarity 0.85)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.85)
• ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agents (NeurIPS 2025 · similarity 0.85)
• SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents (ICLR 2026 · similarity 0.85)