Research map / Multi-agent systems and games / LLM agents

Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formalizes human-centric open-future task discovery, builds HOTD-Bench (~2K+ video clips from Toyota Smarthome Untrimmed and Charades) with LLM-simulation and label-based evaluation, and proposes CMAST, a training-free multi-agent LMM system with a search tree for proposing helpful tasks.

Problem

Discovering tasks that assist humans across multiple concurrent, uncertain future branches from video, where existing LMMs propose tasks mostly for fixed goals and are hard to evaluate exhaustively.

Approach

Model open-future procedures as an explicit search tree of candidate next actions, expanded and pruned by a collaborative multi-agent LMM pipeline, and evaluate helpfulness with an LLM simulator that compares human action costs with and without the robot task instead of exhaustive annotation.

Key results

• CMAST reaches a Valid Task Ratio of 71.7% at 20s on the TSU subset by simulation, versus 58.4% for the best baseline (InternVL2-8B).
• Replacing the search tree with a single LMM agent reduces Valid Task Ratio by 37%.
• Under label-based evaluation, CMAST reaches 38.7% Valid Task Ratio on TSU (vs. 20.6% for Video-LLaVA), and CMAST variants improve their base LMMs' Valid Task Ratio by at least 39%.

Surprising findings

• Larger models (e.g., LLaVA-NeXT-Video-34B, Qwen2.5-VL-72B) show no clear advantage over smaller models for task discovery.
• Baseline LMMs show a trade-off where high Valid Task Ratio comes with low Valid Task Count, and vice versa.

Limitations

• Simulator (LLM) judgments of helpfulness are validated only on a small human study of 50 tasks by 5 annotators.
• Cost is defined only in terms of time consumption, a narrow proxy for human effort.
• Label-based ground truth relies on pipeline-generated annotations from future-performed actions, which may be incomplete.

Experimental setup

Datasets: Toyota Smarthome Untrimmed Charades

Benchmarks: HOTD-Bench

Metrics: Valid Task Count Valid Task Ratio

Baselines: Qwen2-VL-7B Qwen2.5-VL-72B InternVL2-8B InternVL2.5-26B Video-LLaVA-7B LLaVA-NeXT-Video-7B LLaVA-NeXT-Video-34B

Builds on: chain-of-thought multi-agent systems DeepSeek-R1 OpenAI o3

Closest papers by meaning

• PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.88)
• Collaborative Tree Search for Enhancing Embodied Multi-Agent Collaboration (CVPR 2025 · similarity 0.87)
• ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.86)
• From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents (ICLR 2026 · similarity 0.86)
• AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents (ICLR 2026 · similarity 0.86)
• Embodied Task Planning via Graph-Informed Action Generation with Large Language Models (ICML 2026 · similarity 0.86)