Research map / Video / Video-language understanding
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniMMI, a streaming video benchmark for interactive OmniLLMs (2,290 questions over 1,121 videos across six subtasks), and M4, a lightweight attention-based, multiplexed streaming model that adds proactive generation and interruption to pretrained VideoLLMs.
Problem
Existing video benchmarks use full or frame-selected video inputs and do not evaluate streaming, proactive, or turn-taking interactive abilities of omni-modal language models.
Approach
Builds a benchmark with streaming state grounding, multi-turn dependency, action planning, speaker identification, proactive alerting and proactive turn-taking tasks; and proposes M4, which uses attention-based highlight spots for proactive alerting, noise/stop detection, and parallel decoding with multiplexed KV-cache masks, fine-tuned on a small video-free instruction set.
Key results
- Gemini-1.5-Pro scores 52.33 on the first SG stage and 38.50 on PT, while GPT-4o reaches 48.67 on the first SG stage; open-source models drop sharply by the third multi-turn stage (most below 5%).
- M4 (Qwen2-7B) reaches 25.50 on PA and 62.00 on PT in the main table, the best open-source results among streaming-capable models.
- M4 achieves 51.74 on VideoMME, versus 43.52 for the Llama3.1-based variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply M4 to streaming state grounding | Qwen2-7B · OmniMMI SG | accuracy (avg.) | Hurt |
| apply M4 to action planning | Qwen2-7B · OmniMMI AP | accuracy | Hurt |
| apply M4 to multi-turn dependency reasoning | Qwen2-7B · OmniMMI MD | accuracy (avg.) | Hurt |
| apply M4 to speaker identification | Qwen2-7B · OmniMMI SI | accuracy | Hurt |
| apply M4 to proactive turn-taking | Qwen2-7B · OmniMMI PT | accuracy | Hurt |
| add interleaved text-image tuning data (ablation) | M4-Qwen · OmniMMI PA | accuracy: 25.00 -> 26.00 | Helped |
| tune with M4-IT (ablation) | M4-Qwen · OmniMMI PA | accuracy: 25.00 -> 25.50 | Helped |
| average attention weights across query tokens (ablation) | M4-Qwen · OmniMMI PA | accuracy: 25.50 -> 6.50 | Hurt |
| add interleaved text-image tuning data (ablation) | M4-Llama · OmniMMI PA | accuracy: 10.50 -> 10.00 | Hurt |
| tune with M4-IT (ablation) | M4-Llama · OmniMMI PA | accuracy: 10.50 -> 11.50 | Helped |
| average attention weights across query tokens (ablation) | M4-Llama · OmniMMI PA | accuracy: 11.50 -> 0.50 | Hurt |
| add data replay (ablation) | LongVA · VideoMME | General: 52.4 -> 52.2 | Hurt |
Surprising findings
- Models that process both audio and visual inputs do not outperform visual-only models, indicating weak modality alignment.
- Increasing model size does not improve streaming performance; models with 7B parameters are comparable to larger ones, and long-context models perform better.
Limitations
- Multi-turn and proactive tasks remain very hard for open-source models, with many scoring near zero on later stages.
- Audio-visual models do not outperform visual-only models, indicating limited modality alignment.
Experimental setup
Datasets: Ego4D COIN Shot2Story20K QVHighlight MLVU LLaVA-NeXT instruction data
Benchmarks: OmniMMI VideoMME
Metrics: accuracy precision IoU cumulative multi-turn accuracy
Baselines: Gemini-1.5-Pro GPT-4o VideoLLaMA2 VITA VideoLLM-online LongVILA MiniOmni2
Builds on: VideoLLM-online multiplexing modeling duplex speech models (mini-Omni) LongVA
Closest papers by meaning
- StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering (NeurIPS 2026 · similarity 0.90)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.89)
- A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.88)
- Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge (ICLR 2025 · similarity 0.88)
- OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities (ICLR 2025 · similarity 0.88)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.