Research map / Video / Video-language understanding
Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RGTR, a DETR-style temporal sentence grounding model whose moment queries are k-means-initialized static/dynamic anchor pairs plus an IoU-aware scoring head, reducing overlapped proposals.
Problem
DETR-based temporal sentence grounding uses learnable moment queries without task-related guidance, which yields overlapped, redundant proposals and inaccurate localization; proposal scores also ignore boundary quality.
Approach
Replace learnable queries with anchor pairs (static and dynamic anchors) initialized from k-means cluster centers of ground-truth moment spans, so each query covers a specific temporal region; score proposals by the product of classification confidence and a predicted IoU.
Key results
- On QVHighlights test, RGTR reaches 67.12 mAP@0.5 and 45.53 mAP avg, outperforming LLMEPET (65.76 and 44.05).
- On TACoS, RGTR achieves the best R@0.3, R@0.5, R@0.7 and mIoU (37.44 mIoU vs 36.55 for LLMEPET).
- Full RGTR improves R1@0.7 by 3.93 and mAP avg by 3.83 over the baseline setting (a) on QVHighlights val.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RGTR instead of LLMEPET | QVHighlights test · QVHighlights test | mAP Avg.: 44.05 -> 45.53 | Helped |
| use RGTR instead of LLMEPET | QVHighlights val · QVHighlights val | mAP Avg.: 46.24 -> 46.95 | Helped |
| add anchor explicit initialization, region-guided attention, and IoU-aware scoring (ablation) | RGTR on QVHighlights val · QVHighlights val | R0.7: 48.97 -> 52.90 | Helped |
| add anchor explicit initialization, region-guided attention, and IoU-aware scoring (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP_avg: 43.12 -> 46.95 | Helped |
| replace random learnable query initialization with k-means initialization (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP_avg: 44.03 -> 46.95 | Helped |
| replace uniform-grid initialization with k-means initialization (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP_avg: 44.93 -> 46.95 | Helped |
| use classification-times-IoU scoring instead of IoU supervision only (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP_avg: 46.54 -> 46.95 | Helped |
| use classification-times-IoU scoring instead of cls + IoU scoring (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP_avg: 46.92 -> 46.95 | No clear effect |
| set NMS threshold to 0.8 rather than 0.5 (ablation) | RGTR on QVHighlights val · QVHighlights val | mAP@0.75: 47.01 -> 48.00 | Helped |
| replace L2 IoU loss with Huber loss (ablation) | RGTR on QVHighlights val · QVHighlights val | R1@0.7: 52.90 -> 51.94 | Hurt |
| replace L2 IoU loss with L1 loss (ablation) | RGTR on QVHighlights val · QVHighlights val | R1@0.7: 52.90 -> 51.89 | Hurt |
| increase the number of moment queries to 20 (ablation) | RGTR · QVHighlights | mAP_avg | Helped |
Surprising findings
- Uniform grid initialization, which is unrelated to dataset distribution, still improves over random learnable queries, suggesting query diversity matters more than distribution match.
Limitations
- Anchor initialization relies on manual k-means clustering of ground-truth spans, which conflicts with end-to-end DETR philosophy and affects performance.
- Gains are smaller on TACoS and Charades-STA, attributed to biased distributions that reduce query diversity.
- Clustering on the training set may degrade performance when test distribution shifts significantly.
Experimental setup
Datasets: QVHighlights Charades-STA TACoS Charades-STA-Len Charades-STA-Mom Charades-CD
Benchmarks: QVHighlights Charades-STA TACoS Charades-STA-Len Charades-STA-Mom Charades-CD
Metrics: R1@0.5 R1@0.7 mAP mAP@0.5 mAP@0.75 mAP avg mIoU HIT@1
Baselines: M-DETR QD-DETR UniVTG TR-DETR TaskWeave UVCOM CG-DETR LLMEPET MomentDiff 2D-TAN MMN
Builds on: DETR Moment-DETR QD-DETR DAB-DETR DINO
Closest papers by meaning
- Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism (ICML 2025 · similarity 0.86)
- Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.86)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.86)
- Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (ICLR 2024 · similarity 0.85)
- Enhancing Part-Level Point Grounding for Any Open-Source MLLMs (CVPR 2026 · similarity 0.85)
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.