Research map / Vision-language and multimodal / Multimodal LLMs

PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PhoStream, a mobile-centric streaming benchmark of 5,572 open-ended QA pairs over 578 videos, evaluated with an online inference pipeline and LLM-as-a-Judge, revealing severe early-response bias on forward (wait-until-evidence) questions.

Problem

Existing streaming video benchmarks mostly use multiple-choice questions or short videos and do not cover mobile on-screen and off-screen scenarios, so it is unclear whether MLLMs can decide when to respond in continuous audio-visual streams.

Approach

A mobile-centric open-ended streaming benchmark with Backward, Instant, and Forward temporal tasks, built with a Gemini 3 Pro automated generation pipeline plus two rounds of human review, and scored by timing-aware response windows and an LLM judge that gives zero credit to early or missing responses.

Key results

• Gemini 3 Pro scores 80.83 on Instant and 82.19 on Backward but only 16.40 on Forward, with 79.12% early responses on Forward questions.
• Qwen3-Omni-30B-A3B has a 97.89% early response rate on Forward questions, scoring 1.26 on Forward.
• Doubao-Seed-1.6 achieves the best Forward score (44.26) with 29.76% early responses; the human test on 200 videos reproduces these trends.

Surprising findings

• Stronger Instant and Backward performance tends to correlate with more early responses on Forward questions.
• Adding audio improves Instant and Backward scores but slightly lowers Forward scores and raises early responses for Gemini 3 Pro and Qwen3-Omni.

Limitations

• Forward-task scores depend on a fixed 2-second response window and timing rules, which may penalize otherwise valid answers.
• LLM-as-a-Judge scoring relies on a judging model whose identity is not clearly specified in the text, and human test covers only a 200-video subset.

Experimental setup

Datasets: YouTube vlogs YouTube phone tutorials self-recorded phone screen recordings EgoBlind

Benchmarks: PhoStream

Metrics: LLM-judged score (0-100) Early Response rate No Response rate Partly Correct rate

Baselines: Gemini 3 Pro Doubao-Seed-1.6 Doubao-Seed-1.8 Qwen2.5-Omni-7B Qwen3-VL-8B Qwen3-VL-30B-A3B Qwen3-Omni-30B-A3B Dispider VideoLLM-online-8B MMDuet2

Builds on: ProactiveVideoQA StreamingBench OVO-Bench OmniMMI

Closest papers by meaning

• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
• MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.89)
• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs (NeurIPS 2026 · similarity 0.88)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
• StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.87)
• MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.87)