Research map / Multi-agent systems and games / LLM agents

ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ProAct-75, a 75-task video benchmark with step-level annotations and task graphs for proactive response, and ProAct-Helper, a LoRA-tuned Qwen2.5-VL with hierarchical binding loss and entropy-driven thread-aware action selection.

Problem

Proactive agents must continuously monitor video to decide when to intervene and which action to take under task-graph precedence constraints, but existing benchmarks lack task graphs encoding dependencies and parallel threads.

Approach

Combines a multi-source benchmark with explicit AND/OR task DAGs and a MLLM framework that binds trigger, task, and step representations via contrastive hierarchical binding, plus selects robot actions by minimizing human-robot thread-mixing entropy over legal graph actions.

Key results

• ProAct-Helper (7B) reaches trigger mF1 76.56 and task F1 69.20, improving trigger mF1 by 6.21% over... (paper reports 6.21% gain on trigger detection mF1)
• On text-only full-trajectory simulation, ProAct-Helper (7B) achieves SS 0.361 and PA 19.41%, versus Gemini-2.5-Pro's best closed-source baseline.
• Relative to Gemini-2.5-Pro, ProAct-Helper (7B) improves task F1 by 17.09% and step F1 by 11.72%.

Surprising findings

• Closed-source LLMs get lower parallel-action ratios than the simple Greedy strategy, suggesting common-sense decisions avoid parallel threads.
• Gemini-2.5-Pro shows high trigger over-triggering (FPR 77.4%) while GPT-4o misses many triggers (FNR 59.2%).

Limitations

• Cross-view and cross-actor generalization drops substantially (e.g., Ego-Exo4D-style other-view task F1 falls; actor-split step F1 drops from 89.97 to 34.97).
• Future action prediction still shows hallucinations (empty or out-of-vocabulary sequences).
• Planner is a lightweight one-step graph-constrained heuristic; full rollouts need a deadlock-prevention safeguard relaxing graph feasibility.

Experimental setup

Datasets: ProAct-75 Ego-Exo4D COIN UCF-Crime

Benchmarks: ProAct-75

Metrics: mAcc mF1 Acc F1 edit distance Saved Steps Entropy Parallel Actions FPR FNR

Baselines: Qwen3-VL-30B-A3B-Instruct Qwen3-Omni-30B Qwen2.5-VL-32B-Instruct Qwen2.5-VL-3B-Instruct Qwen3-VL-Flash Qwen3-VL-Plus GPT-4o Gemini-2.5-Flash Gemini-2.5-Pro DeepSeek-v3.2 Qwen3-Max Greedy

Builds on: Qwen2.5-VL LoRA Ego-Exo4D COIN UCF-Crime

Closest papers by meaning

• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.91)
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance (ICLR 2025 · similarity 0.91)
• PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety (NeurIPS 2026 · similarity 0.90)
• ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.90)
• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.87)
• ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions (NeurIPS 2025 · similarity 0.87)