Research map / Video / Video-language understanding
Understanding Long Videos with Multimodal Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLM/VLM-based long-video QA can score well with little or no video input, then proposes MVU, a zero-shot framework that adds object-centric video information (captions, detections, trajectories) fused into language with likelihood-selection inference.
Problem
Long-video question answering with LLM-based methods may succeed through world knowledge and language priors rather than actual video understanding; a need exists for efficient, interpretable zero-shot video QA that uses video-specific information.
Approach
Modality-constrained baselines (Just-LLM with text only, Single-Frame-VLM with one frame) reveal strong world-knowledge performance; MVU extracts object categories, spatial locations, and motion trajectories with off-the-shelf image-trained tools and fuses them as natural-language templates, scored with one-forward-pass likelihood selection.
Key results
- Just-LLM (no video input) reaches 45.8% on EgoSchema public subset vs 20% random, and Single-Frame-VLM reaches 55.8%, outperforming an 180-frame SOTA at 50.8%.
- MVU achieves 60.3% on EgoSchema public subset (ES-S) and state-of-the-art zero-shot results on EgoSchema full (61.3% for LVNet+MVU, 37.6% for SF-VLM+MVU) and NExT-QA (73.3% for LVNet+MVU).
- On Open X-Embodiment robotics QA, MVU reaches 30.4% weighted accuracy vs 22.1% random and 28.5% for the LLoVi baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MVU to SF-VLM | SF-VLM · EgoSchema full | accuracy: 36.4 -> 37.6 | Helped |
| Add MVU to LVNet | LVNet · EgoSchema full | accuracy: 61.1 -> 61.3 | Helped |
| Add MVU to SF-VLM | SF-VLM · NextQA, all | accuracy: 54.3 -> 55.4 | Helped |
| Add MVU to LVNet | LVNet · NextQA, all | accuracy: 72.9 -> 73.3 | Mixed |
| Add object-centric MVU to the robotics QA baseline | LLoVi baseline · Open X-Embodiment, total | accuracy: 28.5 -> 30.4 | Helped |
| Add GOI to SF-VLM (ablation) | SF-VLM · EgoSchema-S | accuracy: 55.8 -> 56.4 | Helped |
| Add OSL to MVU with GOI (ablation) | MVU · EgoSchema-S | accuracy: 56.4 -> 58.6 | Helped |
| Add OMT to MVU with GOI and OSL (ablation) | MVU · EgoSchema-S | accuracy: 58.6 -> 60.3 | Helped |
| Replace likelihood selection with answer generation (ablation) | MVU · EgoSchema-S | accuracy: 56.4 -> 60.3 | Hurt |
| Replace likelihood selection with answer generation (ablation) | MVU · NextQA-T | accuracy: 55.3 -> 55.4 | No clear effect |
| Use likelihood selection instead of answer generation (ablation) | MVU · Inference time | inference time: 12.7 -> 2.42 | Helped |
| Include answer candidates in the prompt (ablation) | Likelihood selection · EgoSchema-S | accuracy: 58.2 -> 60.3 | Helped |
Surprising findings
- A text-only LLM with no visual input beats random by +25.8% on EgoSchema-S, indicating many video QA questions are answerable from world knowledge.
- Single-frame VLM with one center frame performs on par with a 180-frame state-of-the-art approach on EgoSchema, and baseline accuracy drops when frame descriptions increase from 16 to 180.
Limitations
- Modality-constrained baselines reveal that benchmarks may be answerable without video, questioning benchmark validity.
- Robotics domain results show near-random performance for text-only and single-frame variants, and MVU gains are modest on some OpenX subsets.
- Full MVU results on EgoSchema rely on baselines with closed ~1.8T-parameter LLMs; the open-model variant is much weaker (37.6%).
Experimental setup
Datasets: EgoSchema NExT-QA Open X-Embodiment
Benchmarks: EgoSchema NExT-QA Open X-Embodiment ActivityNet-QA
Metrics: top-1 accuracy inference time per sample
Baselines: LLoVi LVNet VideoTree VideoAgent InternVideo2 Tarsier SeViLA MC-ViT-L Just-LLM Single-Frame-VLM
Builds on: likelihood selection / cloze prompting LLoVi LVNet
Closest papers by meaning
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.92)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.90)
- TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations (AAAI 2025 · similarity 0.90)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.90)
- MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.