Research map / Video / Video-language understanding
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes whether video MLLMs use motion for referring video segmentation by building a motion-centric benchmark (MoCentric-Bench) with static-keyframe and reversed-video multi-video layouts, plus single-image baselines and a LoRA-adapted Sa2VA.
Problem
Referring video segmentation benchmarks (e.g., MeVIS motion expressions) can often be solved from a single static frame, so it is unclear whether video MLLMs truly use motion for pixel-level visual grounding.
Approach
Four motion-centric probes (single static keyframe, reversed video with reversed expression, and multi-video layouts combining original with keyframe or reverse) expose that state-of-the-art grounding models rely on static cues; a motion-centric LoRA adaptation of Sa2VA's vision encoder with synthetic multi-video training data improves robustness on the probes.
Key results
- Single-image baseline MLLM+SAM 2.0† reaches 71.7 J&F on RefDAVIS-17 and 57.4 J&F on MeVIS val_u, outperforming VideoGLAMM.
- On MoCentric-Bench val_u & Single frame, performance drops by roughly half for all methods (e.g., Sa2VA 58.9 to 28.5 J&F; VideoGLAMM 48.2 to 21.6).
- Motion-centric adapted Sa2VA☆ reaches 60.5 J&F on val_u and 31.1 on val_u & Single frame, and 65.1 on Reverse, the best among compared methods on MoCentric-Bench.
Surprising findings
- Almost half (380 of 793) of MeVIS motion expressions produce false positives on the static keyframe, indicating that a single frame suffices for many motion expressions.
- Existing video MLLM grounding methods degrade sharply under single-frame and reversed multi-video layouts, suggesting they rely on static appearance cues rather than motion.
Limitations
- Motion-centric adaptation only partially closes the gap between multi-video layout and original video (about 1% improvement), leaving substantial room for improvement.
- The reverse-expression probe is limited to 32 videos and 152 pairs after manual filtering, and the single-frame metric can favor background-predicting models.
- Probe construction is empirical rather than grounded in a mathematical framework for motion-centric data collection.
Experimental setup
Datasets: MeVIS RefDAVIS17 Ref-YouTubeVOS ReasonVOS
Benchmarks: RefDAVIS17 MeVIS MoCentric-Bench
Metrics: J F J&F
Baselines: LMPM VideoGLAMM Sa2VA LISA TrackGPT VISA ReferFormer
Builds on: Sa2VA SAM 2 PixFoundation LoRA
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.91)
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.89)
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.89)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.