Research map / Vision-language and multimodal / Multimodal LLMs

IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces IPIBench, a streaming-video benchmark for multi-turn proactive monitoring, proactive task management, and interleaved reactive-proactive requests, and IPI-Agent, a training-free agent with an interaction-control policy and temporal gating that improves several MLLMs.

Problem

Existing streaming video benchmarks evaluate reactive or proactive behavior only in isolated single-turn settings, not dynamic multi-turn streams where users add, modify, or cancel proactive requests alongside reactive queries.

Approach

A multi-turn benchmark covering proactive monitoring, task management (cancel/modify/multi-task), and interleaved reactive-proactive requests, plus a training-free agent that routes intents into persistent proactive memory and gates proactive responses using embedding similarity changes over sliding-window frames.

Key results

• IPI-Agent (Gemini 3 Pro) raises Proactive Monitoring average from 28.93 to 40.08 and Proactive Task Management average from 20.22 to 38.72.
• Best open-source model Qwen3.5-Plus reaches 28.28 average on Interleaved Reactive-Proactive Requests; human level is 96.67.
• IPI-Agent with Qwen3-VL-8B improves its Proactive Monitoring average from 23.96 to 29.03 and Task Management average from 9.35 to 26.74.

Surprising findings

• Pure embedding-based semantic similarity triggering (Qwen3-VL-Embedding) performs markedly worse than the full agent, suggesting semantic similarity alone is insufficient.
• Adding an explicit reminder instruction substantially improves Reactive-under-Proactive performance for all models, indicating models lose proactive objectives without guidance.

Limitations

• Gating thresholds are chosen empirically per base model.
• Even the best model remains far below human-level performance on all task categories.
• Proactive timing errors are dominated by early triggering for some models and late triggering for others.

Experimental setup

Datasets: Ego4D EgoTracks QA-Ego4D RoadTextVQA COIN Charades-STA Oops QVHighlights AVA YouCook2 THUMOS14

Benchmarks: IPIBench

Metrics: accuracy (%)

Baselines: Gemini 3 Pro Gemini 2.5 Pro GPT-5.4 GPT-4o LLaVA-OneVision-7B InternVL3-8B Qwen3-VL-8B Qwen3.5-Plus GLM-4.6V VideoLLM-online-8B Dispider Flash-VStream-7B

Builds on: Qwen3-VL-Embedding

Closest papers by meaning

• ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence On Mobile Devices (CVPR 2026 · similarity 0.87)
• ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response (ICML 2026 · similarity 0.86)
• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.85)
• RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video (NeurIPS 2025 · similarity 0.85)
• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.85)
• Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.84)