Research map / Video / Video-language understanding
Multi-Scale Contrastive Learning for Video Temporal Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-scale and cross-scale contrastive learning over intermediate feature-pyramid levels of a video temporal grounding encoder, using query-centric moment sampling without augmentation or memory banks.
Problem
Long-range moments in feature-pyramid temporal grounding models lose information through downsampling, degrading localization for long moments; prior contrastive methods use moment-query pairs and augmentations or memory banks.
Approach
Draw positive and negative moment samples directly from the encoder's own multi-level feature pyramid for moments tied to a common query, and apply within-scale contrastive loss plus a cross-scale loss that anchors lowest-level moment features to higher-level query-related moments.
Key results
- On TACoS at tIoU=0.5, the model reaches R@1 47.04% and R@5 73.55%, beating SnAG (44.86 / 70.66).
- On Charades-STA at tIoU=0.7, R@1 47.03% and R@5 72.53%, exceeding SSRN by 4.38 and 7.04 points.
- On Ego4D-NLQ with SlowFast+BERT features, average R@1 is 9.49 vs SnAG's 8.33, and average R@5 is 25.06 vs 23.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-scale contrastive learning | SnAG · Ego4D-NLQ, SlowFast+BERT | R@1, tIoU=0.5: 6.83 -> 7.22 | Helped |
| Add multi-scale contrastive learning | SnAG · Ego4D-NLQ, EgoVLP | R@1, tIoU=0.5: 10.78 -> 11.27 | Helped |
| Add multi-scale contrastive learning | SnAG · MAD | R@1, tIoU=0.5: 5.55 -> 6.92 | Helped |
| Add multi-scale contrastive learning | SnAG · TACoS | R@1, tIoU=0.5: +2.18% R@1 | Helped |
| Add multi-scale contrastive learning | SnAG backbone · ActivityNet-Captions | R@1, tIoU=0.7: 33.15 -> 33.56 | Helped |
| Add multi-scale contrastive learning | SnAG · Charades-STA | R@1, tIoU=0.7: 42.65 -> 47.03 | Helped |
| Use data augmentation instead of direct feature-space sampling (ablation) | Multi-scale contrastive learning on TACoS · TACoS | R@1, tIoU=0.5: 47.04 -> 45.46 | Hurt |
| Use a memory bank instead of direct feature-space sampling (ablation) | Multi-scale contrastive learning on TACoS · TACoS | R@1, tIoU=0.5: 47.04 -> 46.62 | Hurt |
| Add within-scale contrastive learning (ablation) | Multi-scale contrastive learning · TACoS | R@1, tIoU=0.5: 46.00 -> 47.04 | Helped |
| Add cross-scale contrastive learning (ablation) | Multi-scale contrastive learning · TACoS | R@1, tIoU=0.5: 45.85 -> 47.04 | Helped |
| Use query-query association instead of query-centric moment-moment association (ablation) | Multi-scale contrastive learning · TACoS | R@1, tIoU=0.5: 47.04 -> 45.06 | Hurt |
| Use moment-query association instead of query-centric moment-moment association (ablation) | Multi-scale contrastive learning · TACoS | R@1, tIoU=0.5: 47.04 -> 46.24 | Hurt |
Surprising findings
- Moment-moment association within the same video outperforms moment-query contrastive learning, but query-query association performs worst, which does not improve moment representations.
Limitations
- Gains on ActivityNet-Captions are small (e.g., +0.41 R@1 over SSRN at tIoU 0.7), and the method is built on the SnAG backbone.
- Ablations are run only on TACoS.
Experimental setup
Datasets: Ego4D-NLQ MAD TACoS ActivityNet-Captions Charades-STA
Benchmarks: Ego4D-NLQ MAD TACoS ActivityNet-Captions Charades-STA
Metrics: Recall@K at tIoU thresholds (R@1, R@5, R@10, R@50)
Baselines: VSL-Net VLG-Net Moment-DETR CONE MMN SSRN G2L SOONet MESM Contrastive-MSAT UVCOM SnAG
Builds on: SnAG focal loss Distance-IoU loss Soft-NMS center sampling
Closest papers by meaning
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.87)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.87)
- Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.86)
- Bridging the Gap: A Unified Video Comprehension Framework for Moment Retrieval and Highlight Detection (CVPR 2024 · similarity 0.86)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.