Research map / Vision-language and multimodal / Cross-modal learning
TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TR-DETR, a DETR-based joint moment retrieval and highlight detection model that adds local-global multi-modal alignment, query-guided visual feature refinement, and a bidirectional task cooperation module (HD2MR, MR2HD) to exploit task reciprocity.
Problem
Joint video moment retrieval (MR) and query-based highlight detection (HD) are typically handled by DETR-based models with two isolated task heads, which underutilize the reciprocal relationship between the tasks and suffer from modality gaps between visual and textual features.
Approach
Align visual and textual features locally (clip-word similarity regularizer) and globally (contrastive loss) before interaction, refine visual features with query guidance, and let HD scores inform MR while retrieved moments refine HD scores via a task cooperation module.
Key results
- On QVHighlights test (video only), TR-DETR reaches R1@0.7 of 48.96 and HD mAP of 39.91, above QD-DETR's 44.98 and 38.64.
- On TVSum (video only), TR-DETR reaches average top-5 mAP 88.1 versus QD-DETR's 85.0.
- On the QVHighlights val ablation, the full model (LGAM+VFR+MR2HD+HD2MR) improves average MR mAP from 38.03 (baseline) to 45.09.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TR-DETR instead of QD-DETR | QVHighlights test (video features) · QVHighlights test | R1@0.7: 44.98 -> 48.96 | Helped |
| use TR-DETR instead of QD-DETR | QVHighlights test (video features) · QVHighlights test | mAP@0.75: 39.88 -> 43.73 | Helped |
| use TR-DETR instead of QD-DETR | QVHighlights test (video and audio features) · QVHighlights test | R1@0.7: 45.10 -> 47.67 | Helped |
| use TR-DETR instead of QD-DETR | QVHighlights test (video and audio features) · QVHighlights test | mAP@0.75: 40.10 -> 42.98 | Helped |
| use TR-DETR instead of QD-DETR | Charades-STA test with I3D features · Charades-STA test | R1@0.5: 50.67 -> 55.51 | Helped |
| use TR-DETR instead of QD-DETR | Charades-STA test with I3D features · Charades-STA test | R1@0.7: 31.02 -> 33.66 | Helped |
| use TR-DETR instead of QD-DETR | TVSum video features · TVSum val | top-5 mAP (Avg): 85.0 -> 88.1 | Helped |
| add LGAM (ablation) | Moment-DETR baseline · QVHighlights val | R1@0.7: 42.35 -> 44.97 | Helped |
| add VFR (ablation) | Moment-DETR baseline · QVHighlights val | R1@0.7: 42.35 -> 47.61 | Helped |
| add MR2HD (ablation) | Moment-DETR baseline · QVHighlights val | R1@0.7: 42.35 -> 42.71 | Helped |
| add HD2MR (ablation) | Moment-DETR baseline · QVHighlights val | R1@0.7: 42.35 -> 42.26 | Hurt |
| add both MR2HD and HD2MR (ablation) | Moment-DETR baseline · QVHighlights val | R1@0.7: 42.35 -> 44.71 | Helped |
Surprising findings
- Adding audio features by direct concatenation slightly degraded some QVHighlights metrics, which the authors attribute to misaligned multi-modal features.
- On Charades-STA with SlowFast+CLIP features, TR-DETR is slightly below UniVTG on R1@0.5 and R1@0.7.
Limitations
- Audio modality is not used effectively; direct concatenation can hurt performance.
- Performance drops on Charades-STA with weaker GloVe/VGG features due to difficulty aligning features from different-scale extractors.
- Hyperparameters and results on some settings (e.g., setting h) show mutual degradation of tasks when modules are combined without alignment.
Experimental setup
Datasets: QVHighlights Charades-STA TVSum AudioSet
Benchmarks: QVHighlights Charades-STA TVSum
Metrics: R1@0.5 R1@0.7 mAP HIT@1 top-5 mAP
Baselines: Moment-DETR QD-DETR UniVTG UMT MH-DETR XML XML+ CAL MCN MINI-Net TCG Joint-VA
Builds on: Moment-DETR QD-DETR UMT DAB-DETR
Closest papers by meaning
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.90)
- Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.89)
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.87)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.86)
- CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer (AAAI 2025 · similarity 0.86)
- When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.