Research map / Video / Video-language understanding
Number it: Temporal Grounding Videos like Flipping Manga
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Number-Prompt (NumPro), overlaying frame numbers on video frames so Vid-LLMs can ground temporal events via OCR-style reading, both training-free and with LoRA fine-tuning on a NumPro-augmented dataset (NumPro-FT).
Problem
Video Large Language Models can understand video content but struggle to output precise start and end timestamps for video temporal grounding (moment retrieval and highlight detection).
Approach
Overlay unique numerical identifiers on each video frame (red, bottom-right, font size 40, chosen via CLIP-based COCO experiments) so the Vid-LLM can output frame-number boundaries directly, requiring no new tokens or architecture changes.
Key results
- NumPro-FT on LongVA-7B-DPO surpasses prior SOTA by 6.1% mIoU on ActivityNet and 7.7% mIoU on Charades-STA, and by 8.5% mAP on QVHighlights.
- Training-free NumPro raises Qwen2-VL-7B mIoU on Charades-STA from 7.9 to 38.5, averaging a 24.7% mIoU gain across datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NumPro at inference | GPT-4o · ActivityNet | mIoU: 23.7 -> 33.6 | Helped |
| Add NumPro at inference | Qwen2-VL-7B · Charades-STA | mIoU: 7.9 -> 38.5 | Helped |
| Add NumPro at inference | LongVA-7B-DPO · ActivityNet | mIoU: 8.2 -> 15.2 | Helped |
| Fine-tune with NumPro-augmented data instead of conventional fine-tuning (ablation) | LongVA-7B-DPO · ActivityNet | mIoU: 29.0 -> 38.8 | Helped |
| Use font size 20 instead of 40 (ablation) | NumPro on Qwen2-VL-7B · Charades-STA | mIoU: 38.5 -> 34.6 | Hurt |
| Use font size 60 or 80 instead of 40 (ablation) | NumPro on Qwen2-VL-7B · Charades-STA | mIoU: 38.5 -> 37.1; 36.9 | Hurt |
| Place numbers at top left, top right, or center instead of bottom right (ablation) | NumPro on Qwen2-VL-7B · Charades-STA | mIoU: 38.5 -> 35.8; 36.8; 34.1 | Hurt |
| Place numbers at bottom left instead of bottom right (ablation) | NumPro on Qwen2-VL-7B · Charades-STA | mIoU: 38.5 -> 39.3 | Helped |
| Use blue, black, or green instead of red numbers (ablation) | NumPro on Qwen2-VL-7B · Charades-STA | mIoU: 38.5 -> 36.6; 36.6; 36.0 | Hurt |
Surprising findings
- Directly overlaying timestamps (e.g., '10.5s') on frames performs worse than overlaying frame numbers.
- Qwen2-VL-72B moment retrieval on ActivityNet rises from 1.0 to 25.5 mIoU with NumPro, showing near-zero baselines recovered without training.
Limitations
- Design of NumPro chosen heuristically on COCO with CLIP, which is only approximate for Vid-LLMs.
- Fine-tuned results rely on a curated 220k-sample dataset and LoRA training on 8 H800 GPUs.
- General video-QA gains are not reported; NumPro minimally affects general QA only on VideoInstruct metrics.
Experimental setup
Datasets: DiDeMo ActivityNet Captions VTimeLLM dataset MSCOCO VideoInstruct Charades-STA ActivityNet QVHighlights MVBench VideoMME
Benchmarks: Charades-STA ActivityNet QVHighlights VideoInstruct MVBench VideoMME
Metrics: mIoU R@0.3 R@0.5 R@0.7 mAP HIT@1 CI/DO/CU/TU/CO scores
Baselines: TimeChat VTimeLLM VTG-LLM HawkEye Momentor GroundingGPT LITA GPT-4o
Builds on: VTimeLLM LongVA Qwen2-VL Set-of-Mark prompting
Closest papers by meaning
- VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding (AAAI 2025 · similarity 0.90)
- TRACE: Temporal Grounding Video LLM via Causal Event Modeling (ICLR 2025 · similarity 0.89)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.88)
- Temporal Sentence Grounding with Relevance Feedback in Videos (NeurIPS 2024 · similarity 0.88)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.87)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.