Research map / Video / Video-language understanding
Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TaskWeave, a task-driven top-down framework for joint moment retrieval and highlight detection with a task-decoupled unit, inter-task mask-based feedback, and a task-dependent uncertainty-weighted loss.
Problem
Joint moment retrieval (MR) and highlight detection (HD) models follow a data-driven bottom-up paradigm with a shared backbone, which overlooks task-specific characteristics and interplay between the tasks, limiting performance.
Approach
A task-decoupled unit (shared expert plus MR-specific and HD-specific experts combined by element-wise product), an inter-task feedback mechanism that converts one task's predictions into masks guiding the other task's decoder, and a learnable uncertainty-based joint loss replacing manually tuned task weights.
Key results
- On QVHighlights val, TaskWeave reaches 50.06 MR R1@0.7 and 45.38 average mAP, with 8.72% Avg mAP improvement over prior SOTA in MR.
- On TVSum, TaskWeave achieves 87.3 average Top-5 mAP, a 2.71% improvement over the previous SOTA (UniVTG non-pretrained was not best; QD-DETR 85.0).
- On Charades-STA, TaskWeave reaches 56.51 R1@0.5 with VGG features, a +7.09% gain over the best compared joint method in R1@0.5 and +8.13% in R1@0.7 per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TaskWeave instead of QD-DETR | Moment retrieval on QVHighlights · QVHighlights val split | Avg. mAP: 41.22 -> 45.38 | Helped |
| Use TaskWeave instead of QD-DETR | Moment retrieval on Charades-STA · Charades-STA test split (VGG) | R1@0.5: 52.77 -> 56.51 | Helped |
| Use TaskWeave instead of QD-DETR | Moment retrieval on Charades-STA · Charades-STA test split (I3D) | R1@0.7: 31.02 -> 31.4 | Helped |
| Use TaskWeave instead of QD-DETR | Highlight detection on QVHighlights · QVHighlights val split | HIT@1: 64.19 -> 63.68 | Hurt |
| Add the task-decoupled unit (ablation) | TaskWeave · QVHighlights val split | Avg. mAP: 41.0 -> 43.24 | Helped |
| Add inter-task feedback (ablation) | TaskWeave with the task-decoupled unit · QVHighlights val split | Avg. mAP: 43.24 -> 45.12 | Helped |
| Add the task-dependent joint loss (ablation) | TaskWeave with the task-decoupled unit and inter-task feedback · QVHighlights val split | Avg. mAP: 45.12 -> 45.38 | Helped |
| Use CNN MR expert and identity HD expert instead of identity experts for both tasks (ablation) | TaskWeave task-decoupled unit · QVHighlights val split | Avg. mAP: 40.23 -> 45.12 | Helped |
| Replace the task-dependent joint loss with summed task losses (ablation) | TaskWeave · QVHighlights val split | Avg. mAP: 45.38 -> 44.27 | Hurt |
| Replace the task-dependent joint loss with a weighted sum (ablation) | TaskWeave · QVHighlights val split | Avg. mAP: 45.38 -> 43.21 | Hurt |
| Use highlightness-guided feedback only instead of moment-guided feedback only (ablation) | TaskWeave inter-task feedback · QVHighlights val split | Avg. mAP: 45.38 -> 44.78 | Hurt |
| Apply both feedback directions simultaneously instead of moment-guided feedback only (ablation) | TaskWeave inter-task feedback · QVHighlights val split | Avg. mAP: 45.38 -> 45.1 | Hurt |
Surprising findings
- Transformer experts underperform simpler CNN or identity experts in the task-decoupled unit, which the authors attribute to high computational complexity.
- Bidirectional feedback (Bi-MRHD) gives limited gain compared to single-direction feedback.
Limitations
- HD results on QVHighlights are second-best, behind UMT which uses audio features.
- Feedback is only beneficial when applied after half of training; direction-specific gains vary across feedback combinations.
- Results rely on pretrained frozen CLIP/SlowFast/I3D backbones and dataset-specific feature choices.
Experimental setup
Datasets: QVHighlights TVSum Charades-STA
Benchmarks: QVHighlights TVSum Charades-STA
Metrics: R1@0.5 R1@0.7 mAP Avg mAP HIT@1 Top-5 mAP
Baselines: Moment-DETR UMT QD-DETR EaTR UniVTG SL-Module 2D-TAN VSLNet MAN
Builds on: Moment-DETR QD-DETR UMT DETR PLE
Closest papers by meaning
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.90)
- TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection (AAAI 2024 · similarity 0.89)
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.87)
- Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection (AAAI 2025 · similarity 0.86)
- When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions (NeurIPS 2025 · similarity 0.85)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.