Research map / Video / Video-language understanding
Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UVCOM, a unified transformer framework for moment retrieval and highlight detection that uses a Comprehensive Integration Module (RBF EM-attention, bidirectional modality random walk, global knowledge accumulation) plus multi-aspect contrastive learning.
Problem
Moment retrieval (MR) and highlight detection (HD) are usually treated as one generic video grounding problem, ignoring that MR needs local relation perception while HD needs global context, which hurts consistent performance on both tasks.
Approach
Designs task-motivated modules: RBF-kernel EM attention for intra-modality aggregation, a bidirectional modality random walk for clip-word interaction with temporal convolution (local relation perception), and a moment-aware intermediary that accumulates global video context for HD, supervised by clip-text alignment and video-linguistic contrastive losses.
Key results
- On TACoS, UVCOM reaches 23.32 R1@0.7 vs. UniVTG's 17.35 (about +6 points).
- On QVHighlights test, UVCOM with audio achieves 47.47 R1@0.7 for MR and 64.79 HIT@1 for HD (UVCOM† variant), outperforming QD-DETR.
- On YouTube Highlights, UVCOM2 reaches 77.4% Avg. mAP and on TVSum 86.3 Top-5 mAP, both state-of-the-art in the paper's comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CIM to the joint framework (ablation) | UVCOM · QVHighlights val | R1@0.7: 44.06 -> 51.81 | Helped |
| add MCL to the joint framework (ablation) | UVCOM · QVHighlights val | R1@0.7: 48.77 -> 51.81 | Helped |
| add DBIA to CIM (ablation) | UVCOM · QVHighlights val | R1@0.7: 50.0 -> 51.81 | Helped |
| add LRP to CIM (ablation) | UVCOM · QVHighlights val | R1@0.7: 50.0 -> 51.81 | Helped |
| add GKA to CIM (ablation) | UVCOM · QVHighlights val | HIT@1: 64.26 -> 63.29 | Mixed |
| use RBF EM-Att instead of average aggregation (ablation) | DBIA · QVHighlights val | R1@0.7: 49.87 -> 51.81 | Helped |
| use RBF EM-Att instead of K-Means (ablation) | DBIA · QVHighlights val | R1@0.7: 48.39 -> 51.81 | Helped |
| add the RBF kernel to EM-Att aggregation (ablation) | DBIA · QVHighlights val | R1@0.7: 50.26 -> 51.81 | Helped |
| use BMRW instead of cross attention (ablation) | LRP · QVHighlights val | R1@0.7: 49.87 -> 51.81 | Helped |
| use UVCOM instead of UniVTG | video moment retrieval · TACoS test | R1@0.7: +5.97% | Helped |
| use UVCOM instead of QD-DETR | video highlight detection · QVHighlights test | HIT@1: +2% | Helped |
Surprising findings
- Cross-attention replacement of BMRW reduces performance by about 2% in the ablation, while the paper attributes gains to attention drift in cross-attention.
- TVSum gains are inconsistent across domains due to small and inconsistent scoring criteria, per the authors.
Limitations
- Audio features are handled simply without a dedicated design, as the authors acknowledge.
- Gains on TVSum are inconsistent across domains due to small, inconsistent scoring criteria.
Experimental setup
Datasets: QVHighlights Charades-STA TACoS YouTube Highlights TVSum
Benchmarks: QVHighlights Charades-STA TACoS YouTube Highlights TVSum
Metrics: R1@0.5 R1@0.7 mAP HIT@1 Top-5 mAP
Baselines: Moment-DETR UMT UniVTG MH-DETR QD-DETR EaTR 2D-TAN VSLNet LIM-S SL-Module MINI-Net TCG
Builds on: Moment-DETR QD-DETR UMT UniVTG EM Attention
Closest papers by meaning
- TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (AAAI 2024 · similarity 0.90)
- Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.90)
- Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection (AAAI 2025 · similarity 0.89)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.88)
- Multi-Scale Contrastive Learning for Video Temporal Grounding (AAAI 2025 · similarity 0.86)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.