Research map / Video / Video-language understanding
Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Momentor, a 7B Video-LLM with a Temporal Perception Module (continuous temporal token space plus neighboring token propagation) trained via grounded event-sequence modeling on Moment-10M, a 10M-instruction segment-level video dataset generated automatically.
Problem
Existing Video-LLMs capture only coarse video-level semantics and cannot precisely localize or reason about specific temporal segments in untrimmed videos, partly because they represent time poorly and lack segment-level instruction data.
Approach
Represent timestamps with learnable temporal tokens arranged in a continuous interpolated embedding space, smooth them with neighboring token propagation, inject them into frame features, pretrain on event-sequence decoding, and instruction-tune on a large automatically generated segment-level dataset (Moment-10M).
Key results
- Momentor (7B) reaches 42.9 R@0.3 and 29.3 mIoU on ActivityNet Captions temporal grounding, above Valley's 30.6 R@0.3 and 21.9 mIoU.
- Momentor achieves 24.4 MoF on Breakfast action segmentation versus 11.6 for the best baseline Video-LLaMA.
- Momentor reaches 68.9% accuracy on MSVD-QA, above Valley's 65.4%, but scores 40.8 on ActivityNet-QA, below Valley's 45.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add continuous interpolation (ablation) | Momentor · ActivityNet-Captions | mIoU: 27.6 -> 29.3 | Helped |
| Add neighboring token propagation (ablation) | Momentor · ActivityNet-Captions | mIoU: 25.4 -> 29.3 | Helped |
| Add Grounded Event-Sequence Modeling (ablation) | Momentor · ActivityNet-Captions | CIDEr: 9.8 -> 14.6 | Helped |
| Add cross-segment tasks during fine-tuning (ablation) | Momentor · QVHighlights | mAP: 6.4 -> 7.6 | Helped |
| Add continuous interpolation (ablation) | Momentor · ActivityNet-Captions | CIDEr: 13.1 -> 14.6 | No clear effect |
Surprising findings
- On ActivityNet-QA, Momentor (40.8 accuracy) trails Valley (45.1), so it is not uniformly best on video QA.
- Removing neighboring token propagation lowers all reported metrics, and removing grounded event-sequence modeling sharply reduces dense captioning CIDEr from 14.6 to 9.8.
Limitations
- Momentor does not model spatial information, so instance reference is limited to temporal context.
- Moment-10M is generated with automatic tools (Grounding DINO, Vicuna), so label quality depends on those pipelines.
- Performance gains slow down once training data reaches million-level scale.
Experimental setup
Datasets: Moment-10M YTTemporal-1B Breakfast 50Salads ActivityNet Captions Charades-STA QVHighlights ActivityNet-QA MSRVTT-QA MSVD-QA
Benchmarks: Breakfast 50Salads ActivityNet Captions Charades-STA QVHighlights MSVD-QA MSRVTT-QA ActivityNet-QA
Metrics: MoF F1@{10,25,50} SODA_c CIDEr METEOR R@0.3 R@0.5 R@0.7 mIoU mAP
Baselines: Video-ChatGPT VideoChat Video-LLaMA Valley Vid2Seq
Builds on: Video-ChatGPT Vid2Seq Grounding DINO Vicuna
Closest papers by meaning
- VTimeLLM: Empower LLM to Grasp Video Moments (CVPR 2024 · similarity 0.92)
- On the Consistency of Video Large Language Models in Temporal Comprehension (CVPR 2025 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.90)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.90)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.90)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.