Research map / Video / Video-language understanding
Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DualGround, a dual-branch VTG model that routes the [EOS] token through a dummy-enhanced sentence path and clusters word tokens into phrases for a phrase-clip path, achieving state-of-the-art moment retrieval and highlight detection on QVHighlights and Charades-STA.
Problem
Existing video temporal grounding models treat all text tokens uniformly in cross-modal attention, over-relying on the [EOS] global token and underusing word-level cues needed for fine-grained localization.
Approach
Separate global and local text semantics into two paths: an [EOS]-only sentence path with learnable dummy tokens (Adaptive Cross Attention), and a phrase path that recurrently groups words into N phrases, refines them with slot attention, and aggregates phrase-clip interactions guided by a reconstructed [EOS] token.
Key results
- On QVHighlights test with InternVideo2 features, DualGround reaches R1@0.7 56.94 vs. FlashVTG 53.96 and mAP 52.73 vs. 52.00.
- On QVHighlights val with IV2, DualGround achieves R1@0.5 73.48, R1@0.7 58.97, mAP 53.26 (best across CG-DETR, TR-DETR, FlashVTG).
- On Charades-STA test with IV2, DualGround obtains R1@0.5 70.67 and R1@0.7 50.33, edging out FlashVTG (70.32 / 49.87).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DualGround instead of FlashVTG | QVHighlights test with CLIP + SlowFast · QVHighlights test | R1@0.7: 50.00 -> 51.72 | Helped |
| use DualGround instead of FlashVTG | QVHighlights test with InternVideo2 · QVHighlights test | R1@0.7: 53.96 -> 56.94 | Helped |
| use DualGround instead of FlashVTG | QVHighlights val with InternVideo2 · QVHighlights val | R1@0.7: 56.06 -> 58.97 | Helped |
| use DualGround instead of FlashVTG | Charades test with CLIP + SlowFast · Charades test | R1@0.7: 37.89 -> 38.52 | Helped |
| use DualGround instead of FlashVTG | Charades test with InternVideo2 · Charades test | R1@0.7: 49.87 -> 50.33 | Helped |
| replace addition fusion with Hadamard fusion (ablation) | DualGround · QVHighlights val | R1@0.7: 58.97 -> 55.25 | Hurt |
| replace addition fusion with Gate fusion (ablation) | DualGround · QVHighlights val | R1@0.7: 58.97 -> 58.71 | Hurt |
| replace addition fusion with Concat-mlp fusion (ablation) | DualGround · QVHighlights val | R1@0.7: 58.97 -> 58.19 | Hurt |
| add the DQA loss (ablation) | DualGround with RPG and Slot Attention, without DQA or EOS losses · QVHighlights val | R1@0.7: 56.17 -> 58.97 | Helped |
| add the EOS reconstruction loss (ablation) | DualGround with RPG and Slot Attention, without DQA or EOS losses · QVHighlights val | R1@0.7: 56.17 -> 58.97 | Helped |
| add the DQA loss (ablation) | DualGround with RPG and Slot Attention, without DQA loss · QVHighlights val | R1@0.7: 56.55 -> 58.97 | Helped |
| add the EOS reconstruction loss (ablation) | DualGround with RPG and Slot Attention, without EOS loss · QVHighlights val | VG-Hit@1: 71.77 -> 71.62 | Hurt |
Surprising findings
- With CLIP-based SF+C features, [EOS]-only input underperforms word-only input for all prior models, while with InternVideo2 [EOS]-only matches or beats full input for most prior models.
- The authors report that prior VTG models show high Pearson/Spearman correlation (close to 1 on train) between [EOS] and word-token attention, indicating [EOS] dominance.
Limitations
- Assumes a fixed number of phrases per query, which must be tuned per dataset (N=3 for Charades, N=4 for QVHighlights).
- Does not use audio features, limiting audio-visual grounding scenarios.
Experimental setup
Datasets: QVHighlights Charades-STA TVSum
Benchmarks: QVHighlights Charades-STA TVSum
Metrics: R1@0.5 R1@0.7 mAP mAP@0.5 mAP@0.75 VG-mAP VG-Hit@1 mIoU Hit@1
Baselines: CG-DETR TR-DETR UVCOM R2-Tuning FlashVTG UniVTG QD-DETR UMT LIM-S Trailer SL-Module
Builds on: CG-DETR FlashVTG R2-Tuning LGI Slot Attention
Closest papers by meaning
- Temporal Sentence Grounding with Relevance Feedback in Videos (NeurIPS 2024 · similarity 0.91)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.90)
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.90)
- Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding (AAAI 2024 · similarity 0.90)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.89)
- VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.