Research map / Video / Video-language understanding
Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MVTCL, a weakly supervised temporal article grounding framework that models multi-scale video segments and hierarchical article sentences with a semantic calibration module and multi-scale contrastive loss, reaching new SOTA on WSAG benchmarks.
Problem
Weakly supervised temporal article grounding (WSAG): localizing video segments for all groundable sentences of a multi-scale article without temporal annotations, where sentences and video segments exist at different semantic/temporal scales.
Approach
Multi-Scale Semantic Calibration aggregates visual context at several temporal scales and suppresses low-level sentence attention with high-level summary attention via Hadamard products; a multi-scale contrastive loss selects non-overlapping proposals per scale for inter-video contrastive learning.
Key results
- MVTCL reaches R@50 (IoU=0.1) of 48.54 vs. 40.11 for DualMIL on the WSAG benchmark.
- MVTCL reaches R@100 (IoU=0.1) of 64.62 vs. 54.32 for DualMIL.
- Ablation: removing both MSC and MCL drops R@50 (IoU=0.1) from 48.54 to 26.60.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MVTCL instead of DualMIL | WSAG · CrossTask | R@50 (IoU=0.1): 40.11 -> 48.54 | Helped |
| add MSC and MCL (ablation) | MVTCL · CrossTask | R@50 (IoU=0.1): 26.60 -> 48.54 | Helped |
| add MCL (ablation) | MVTCL · CrossTask | R@50 (IoU=0.1): 37.93 -> 48.54 | Helped |
| add MSC (ablation) | MVTCL · CrossTask | R@50 (IoU=0.1): 36.74 -> 48.54 | Helped |
| use only visual scale G1 instead of all scales (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 48.54 -> 34.99 | Hurt |
| use only visual scale G2 instead of all scales (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 48.54 -> 29.32 | Hurt |
| use only visual scale G3 instead of all scales (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 48.54 -> 32.32 | Hurt |
| use visual scales G1 and G2 only (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 37.93 -> 33.12 | Hurt |
| use visual scales G1 and G3 only (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 37.93 -> 32.62 | Hurt |
| use visual scales G2 and G3 only (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 37.93 -> 32.21 | Hurt |
| remove hierarchical language semantic suppression (ablation) | MSC · CrossTask | R@50 (IoU=0.1): 37.93 -> 27.39 | Hurt |
| use only loss scale G1 instead of all scales (ablation) | MCL · CrossTask | R@50 (IoU=0.1): 48.54 -> 47.56 | Hurt |
Surprising findings
- Zero-shot MIL-NCE with avg-pooling is competitive at higher IoU thresholds, outperforming MVTCL-style baselines at R@50 IoU=0.5 (12.88 vs. MVTCL 12.91 essentially tied) and RC@50 at IoU 0.1 is lower than DualMIL for MVTCL.
Limitations
- Relies on frozen S3D clip features and GloVe word embeddings with a BiLSTM text encoder.
- Random sampling of 20 sentences per training sample and fixed scale settings (G = 1, 2, 4) are hand-chosen design choices.
- Evaluated only on YouwikiHow training and CrossTask test; generalization to other article-video domains is not shown.
Experimental setup
Datasets: YouwikiHow CrossTask
Benchmarks: CrossTask
Metrics: Recall@50 (IoU 0.1/0.3/0.5) Recall@100 (IoU 0.1/0.3/0.5) RC@50 RC@100
Baselines: RandomGuess WSTAN-full WSTAN-base DualMIL MIL-NCE-max MIL-NCE-avg MIL-NCE+WSTAN
Builds on: DualMIL WSTAN MIL-NCE TGA
Closest papers by meaning
- Hierarchical Semantic Correspondence Networks for Video Paragraph Grounding (CVPR 2023 · similarity 0.92)
- Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding (CVPR 2024 · similarity 0.90)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.90)
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.89)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.88)
- Temporal Sentence Grounding with Relevance Feedback in Videos (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.