Research map / Vision-language and multimodal / Cross-modal learning
CDTR: Semantic Alignment for Video Moment Retrieval Using Concept Decomposition Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDTR, a video moment retrieval transformer that decomposes video clips and queries into concept representations and uses them as pseudo-labels with adaptive thresholds for fine-grained video intra-modal and textual-visual contrastive alignment.
Problem
Video moment retrieval struggles with fine-grained semantic understanding, semantic overlap among negative samples, and sparse annotation constraints when aligning video moments with natural language queries.
Approach
Disentangle video clips and query words into a 17,533-word CLIP-embedded concept space, use the decomposed concept activations as pseudo-labels with mean/std adaptive thresholds to pick concept-specific positives and negatives, and apply Hadamard-product concept extraction for contrastive alignment within video and across text-video.
Key results
- On QVHighlights test split, CDTR reaches R1@0.5 65.79 and mAP@0.5 49.60, outperforming UVCOM (63.55, 47.47) with average gain 2.38%.
- On Charades-STA with CLIP+SlowFast features, CDTR achieves R1@0.7 of 50.65 mIoU 60.39 (mIoU, R1@0.5 37.24 ... as reported), surpassing UVCOM on all metrics.
- On TACoS, CDTR leads UVCOM by 3.87% in R1@0.5 (R1@0.5 of 40.26 mIoU vs 36.39 for UVCOM as reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add concept decomposition and its associated loss functions (ablation) | CDTR · QVHighlights val split | R1@0.5: 59.35 -> 62.19 | Helped |
| add video intra-modal alignment (VIA) (ablation) | CDTR · QVHighlights val split | R1@0.5: 62.19 -> 62.74 | Helped |
| add textual-visual cross-modal alignment (TCA) (ablation) | CDTR · QVHighlights val split | R1@0.5: 62.74 -> 63.99 | Helped |
| add sequential attention (SAT) (ablation) | CDTR · QVHighlights val split | R1@0.5: 65.24 -> 68.03 | Helped |
| use Hadamard product for concept extraction instead of vector projection (ablation) | CDTR · QVHighlights val split | R1@0.5: 65.31 -> 68.03 | Helped |
| remove concept extraction (ablation) | CDTR · QVHighlights val split | R1@0.5: 68.03 -> 61.98 | Hurt |
| remove query relevance (ablation) | CDTR · QVHighlights val split | R1@0.5: 65.12 -> 68.03 | Hurt |
| remove semantic mining (ablation) | CDTR · QVHighlights val split | R1@0.5: 66.85 -> 68.03 | Hurt |
| add textual-visual cross-modal alignment (TCA) (ablation) | CDTR · QVHighlights val split | R1@0.5: +1.26% | Helped |
| add video intra-modal alignment (VIA) (ablation) | CDTR · QVHighlights val split | R1@0.5: +0.51% | Helped |
Surprising findings
- On Charades-STA with VGG features, CDTR is only second best in R1@0.7 (36.20) behind BM-DETR (36.24).
Limitations
- Gains over strongest baselines are modest on some metrics (e.g., VGG Charades-STA R1@0.7 second best).
- Relies on fixed pre-extracted CLIP and SlowFast features rather than end-to-end training.
- Concept library is fixed to BERT vocabulary words, and ambiguous pseudo-labels in middle of distribution are discarded.
Experimental setup
Datasets: QVHighlights Charades-STA TACoS
Benchmarks: QVHighlights Charades-STA TACoS
Metrics: R1@0.5 R1@0.7 mAP mIoU
Baselines: UVCOM MESM QD-DETR UniVTG BM-DETR TaskWeave UMT MomentDiff TR-DETR M-DETR
Builds on: DETR QD-DETR CLIP SlowFast
Closest papers by meaning
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.91)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.89)
- VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · similarity 0.88)
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.88)
- Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D (CVPR 2025 · similarity 0.86)
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.