Research map / Video / Video-language understanding
Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Keyword-DETR, which clusters video clips to capture overall context, reweights query keywords by their video-context relevance, and adds keyword-aware contrastive losses for joint moment retrieval and highlight detection.
Problem
Joint video moment retrieval and highlight detection from text queries struggle to determine which query words matter, since word importance depends on the overall video context, and existing text features are extracted independently of the video.
Approach
A video context clustering module (temporally-weighted FINCH) summarizes the video into scene clusters; cosine similarity between clusters and query words yields keyword weights that reweight text features, which then feed keyword-aware contrastive losses (clip-keyword and video-keyword) and cluster-derived cues for the MR and HD heads.
Key results
- On QVHighlights test (video only), the method reaches R1@0.5 66.86 and mAP Avg 45.69, versus 64.66 and 43.18 for the best prior method (TR-DETR/UVCOM).
- On TVSum, the method averages 88.7 top-5 mAP, best in 8 of 10 categories, versus 88.1 for TR-DETR.
- On Charades-STA with SF+C features, R1@0.7 improves to 37.89 from 36.64 (UVCOM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VCKA (ablation) | Method without VCKA or VCP · QVHighlights val | R1@0.5: 66.39 -> 68.90 | Helped |
| Add VCP (ablation) | Method without VCKA or VCP · QVHighlights val | R1@0.5: 66.39 -> 66.90 | Helped |
| Add VCP alongside VCKA (ablation) | VCKA-only method · QVHighlights val | R1@0.7: 52.32 -> 53.35 | Helped |
| Add clip-keyword contrastive loss (ablation) | Method without keyword-aware contrastive losses · QVHighlights val | R1@0.5: 66.90 -> 67.74 | Helped |
| Add video-keyword contrastive loss (ablation) | Method without keyword-aware contrastive losses · QVHighlights val | R1@0.5: 66.90 -> 67.16 | Helped |
| Combine clip-keyword and video-keyword contrastive losses (ablation) | Method with neither keyword-aware contrastive loss · QVHighlights val | mAP Avg.: 45.02 -> 47.69 | Helped |
| Set λkw to 0.1 instead of 0.3 (ablation) | Ours · QVHighlights val | mAP Avg.: 47.69 -> 46.65 | Hurt |
| Set λkw to 0.5 instead of 0.3 (ablation) | Ours · QVHighlights val | mAP Avg.: 47.69 -> 47.68 | No clear effect |
| Set λkw to 0.7 instead of 0.3 (ablation) | Ours · QVHighlights val | mAP Avg.: 47.69 -> 46.89 | Hurt |
| Set λkw to 0.9 instead of 0.3 (ablation) | Ours · QVHighlights val | R1@0.5: 68.97 -> 69.29 | Mixed |
| Use the proposed method instead of TR-DETR | Video-only method · QVHighlights test | mAP Avg.: 42.62 -> 45.69 | Helped |
| Use the proposed method instead of TR-DETR | Method with video and audio · QVHighlights test | HD mAP: 39.90 -> 41.15 | Helped |
Limitations
- Audio is represented by a simplified PANN feature without a dedicated audio-visual framework.
- Training time per iteration is slightly higher than TR-DETR due to the clustering step.
Experimental setup
Datasets: QVHighlights TVSum Charades-STA
Benchmarks: QVHighlights TVSum Charades-STA
Metrics: R1@0.5 R1@0.7 mAP mAP@Avg HIT@1 top-5 mAP
Baselines: M-DETR QD-DETR UniVTG TR-DETR UVCOM UMT 2D-TAN sLSTM LIM-S Trailer SL-Module
Builds on: Moment-DETR FINCH QD-DETR TR-DETR UniVTG
Closest papers by meaning
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.89)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.86)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.86)
- Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D (CVPR 2025 · similarity 0.86)
- TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (AAAI 2024 · similarity 0.86)
- Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.