Research map / Video / Video-language understanding
Universal Video Temporal Grounding with Generative Multi-modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds UniTime, a Qwen2-VL-7B-based video temporal grounding model that interleaves textual timestamp tokens with video frames, uses adaptive frame scaling and coarse-to-fine inference, and reports SOTA grounding and improved long-video VideoQA.
Problem
Temporal grounding of natural-language queries in videos generalizes poorly across domains and durations, and existing MLLM-based grounders are limited by context windows and memory on long videos.
Approach
Interleave textual timestamp tokens before frames (or segments) so the MLLM retrieves timestamps by reading them, combined with adaptive per-frame token budgets (resize short videos, compress long ones), multi-stage coarse-to-fine inference, and video-centric training that packs multiple queries per video.
Key results
- UniTime-SP reaches 61.18 R1@0.3 on TACoS and 54.81 R1@0.5 on ANet-Captions (dataset-specific fine-tuning), outperforming prior SOTA on Ego4D-NLQ, TACoS, Charades-STA and QVHighlights.
- UniTime-Zero improves Charades-STA zero-shot R1@0.5 to 59.09 vs 48.95 for TimeSuite and TACoS R1@0.3 to 50.06 vs 24.59 for Mr.BLIP.
- On VideoQA with grounding-then-answer, UniTime-Full raises QaEgo4D accuracy to 55.51 vs 49.60 uniform sampling and CG-Bench accuracy to 40.30 vs 33.87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add coarse segment retrieval to adaptive scaling and multi-stage inference (ablation) | UniTime · Ego4D-NLQ | R1@.3: 17.91 -> 24.79 | Helped |
| add multi-stage inference to uniform sampling (ablation) | UniTime · Ego4D-NLQ | R1@.3: 14.25 -> 18.42 | Helped |
| add adaptive frame scaling to uniform sampling (ablation) | UniTime · Ego4D-NLQ | R1@.3: 14.25 -> 14.00 | No clear effect |
| replace long-video frame resizing with token compression (ablation) | UniTime · Ego4D-NLQ | R1@.3: 18.53 -> 24.79 | Helped |
| replace short-video frame resizing with token compression (ablation) | UniTime · Ego4D-NLQ | R1@.3: 24.79 -> 24.06 | No clear effect |
| replace dense position encodings with timestamp token insertion (ablation) | UniTime · Charades-STA | R1@.5: 48.44 -> 74.33 | Helped |
| reduce token budget from 16384 to 8192 (ablation) | UniTime · Ego4D-NLQ | mIoU: 17.25 -> 16.69 | Hurt |
| reduce token budget from 16384 to 4096 (ablation) | UniTime · Ego4D-NLQ | mIoU: 17.25 -> 15.96 | Hurt |
| use N_f^short=256 instead of 128, causing more inputs in the 128–256s range to use single-stage processing (ablation) | UniTime · TaCoS 128–256s | mIoU: 54.57 -> 42.90 | Hurt |
| use UniTime moment retrieval before answering VideoQA rather than the UniVTG retrieval baseline | Qwen2-VL-7B VideoQA pipeline · QaEgo4D | Acc.: 50.00 -> 55.51 | Helped |
| use UniTime moment retrieval before answering VideoQA rather than the UniVTG retrieval baseline | Qwen2-VL-7B VideoQA pipeline · CG-Bench | Acc.: 34.87 -> 40.30 | Helped |
| use UniTime moment retrieval before answering VideoQA rather than uniform frame sampling | Qwen2-VL-7B VideoQA pipeline · LongVideoBench | Acc.: 54.82 -> 56.47 | Helped |
Surprising findings
- Off-the-shelf Qwen2.5-VL-7B and Qwen2-VL-7B score very low on Ego4D-NLQ (under 1.2 R1@0.3) without UniTime training, despite strong short-video grounding.
- On ANet-Captions UniTime-Full underperforms prior SOTA (e.g. -0.23 R1@0.5 vs Momentor), which the authors attribute to annotation incompleteness.
Limitations
- Limited to temporal grounding; lacks reasoning and QA ability from training data alone.
- Fixed segment length for long videos lacks flexibility.
- Underperforms prior SOTA on ANet-Captions, partly attributed to annotation issues.
Experimental setup
Datasets: NaQ DiDeMo QuerYD HiREST COIN Momentor YouCook2 Ego4D-NLQ TACoS Charades-STA QVHighlights ActivityNet-Captions
Benchmarks: Ego4D-NLQ TACoS Charades-STA ActivityNet-Captions QVHighlights QaEgo4D CG-Bench MLVU LongVideoBench Charades-CON ActivityNet-CON
Metrics: R1@0.3 R1@0.5 R1@0.7 mIoU IoP IoG accuracy
Baselines: UniVTG Mr.BLIP VTimeLLM TimeChat TimeSuite VTG-LLM Momentor TimeMarker SG-DETR LD-DETR SnAG Qwen2.5-VL-7B
Builds on: Qwen2-VL VTimeLLM Mr.BLIP UniVTG
Closest papers by meaning
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.90)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.89)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.89)
- SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · similarity 0.89)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.