Research map / Video / Video-language understanding
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Motion-Grounded Video Reasoning, a task requiring pixel-level spatiotemporal masks as answers to implicit motion questions, with the GROUNDMORE benchmark and the MORA baseline built on LISA with a temporal [LOC] head.
Problem
Existing video understanding tasks address motion only partially: they lack spatial object context, temporal context, implicit reasoning, or pixel-level output. Current video reasoning segmentation benchmarks do not model temporal localization of motion.
Approach
Frame the motion understanding as a question-answering task whose answer is a spatiotemporal segmentation mask of the motion-related object, and extend LISA with spatiotemporal pooling and a [LOC] token that predicts a binary temporal mask to suppress false temporal activations.
Key results
- MORA fine-tuned reaches 27.15 overall J&F on GroundMoRe, versus 11.17 for PG-Video-LLaVA in the zero-shot comparison table.
- MORA zero-shot achieves 23.13 J&F overall, outperforming the best video reasoning baseline by an average of 11.28 points, and the best two-stage baseline SeViLA+SgMg at 22.34.
- Providing ground-truth answers instead of questions raises J&F by about 14.29 on average for RVOS models, indicating implicit reasoning is a major difficulty.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the temporal localization branch (ablation) | MORA · GROUNDMORE Overall | J&F: 25.62 -> 27.15; 6.0% relative improvement | Helped |
| add the temporal localization branch (ablation) | MORA · GROUNDMORE Causal | J&F: 26.25 -> 27.42 | Helped |
| add the temporal localization branch (ablation) | MORA · GROUNDMORE Sequential | J&F: 24.66 -> 26.07 | Helped |
| add the temporal localization branch (ablation) | MORA · GROUNDMORE Counterfactual | J&F: 24.34 -> 25.56 | Helped |
| add the temporal localization branch (ablation) | MORA · GROUNDMORE Descriptive | J&F: 28.44 -> 29.55 | Helped |
Surprising findings
- Image reasoning segmentation models (LISA, PixelLM) score lower than the referring VOS models, attributed to lack of temporal modeling.
- PG-Video-LLaVA, a video reasoning segmentation model, does not outperform the best RVOS model, likely due to redundant grounding of salient objects.
Limitations
- Overall performance remains low (best around 27 J&F), leaving large room for improvement.
- MORA relies on basic components (LLaVA, spatiotemporal pooling, simple temporal head) that may lose information.
- Frame-by-frame adaptation of image models and mask propagation errors degrade performance on temporal questions.
Experimental setup
Datasets: GroundMoRe Ref-YouTubeVOS MeViS
Benchmarks: GroundMoRe
Metrics: J&F J F
Baselines: ReferFormer SgMg HTR LMPM LISA PixelLM PG-Video-LLaVA VISA ViLA VideoChat2 SeViLA
Builds on: LISA Video-ChatGPT SAM LLaVA MeViS
Closest papers by meaning
- PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? (NeurIPS 2026 · similarity 0.91)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.90)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.89)
- Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning (AAAI 2026 · similarity 0.88)
- TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.88)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.