Research map / Video / Video-language understanding
Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMDist, which trains a multi-modal (RGB + optical flow) teacher with cross-modal mutual learning and distills its temporal boundaries and semantic activation maps into a single-modal RGB student for weakly-supervised temporal video grounding.
Problem
Weakly-supervised temporal video grounding suffers from inadequate supervision because no temporal boundary annotations are available, and multi-modal inputs are impractical at inference time.
Approach
Use multi-modal videos only during training: a teacher learns from RGB and optical flow via cross-modal mutual learning, and a single-modal student is trained with local (semantic activation map) and global (teacher-predicted boundary pseudo-labels) distillation, on top of a local-global contrastive learning baseline.
Key results
- On Charades-STA, MMDist Student reaches R@0.5 of 53.29 vs. 49.24 for the previous best weakly-supervised method CPL.
- MMDist Teacher (RGB+optical flow) reaches R@0.7 of 26.00 on Charades-STA vs. 22.39 for CPL.
- On ActivityNet Captions, MMDist Student reaches R@0.3 of 58.69 vs. 53.67 for CPL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use local-global contrastive learning instead of CPL | MMDist Baseline · Charades-STA | R@0.5: 49.24 -> 51.58 | Helped |
| use local-global contrastive learning instead of CPL | MMDist Baseline · ActivityNet Captions | R@0.3: 53.67 -> 56.92 | Helped |
| add multi-modal video input with cross-modal mutual learning | MMDist Teacher · Charades-STA | R@0.7: 22.39 -> 26.00 | Helped |
| distill teacher knowledge to the single-modal student | MMDist Student · ActivityNet Captions | R@0.3: 56.92 -> 58.69 | Helped |
| add local contrastive learning (ablation) | MMDist Baseline · Charades-STA | R@0.3: 65.83 -> 67.26 | Helped |
| add global contrastive learning (ablation) | MMDist Baseline · Charades-STA | R@0.3: 60.42 -> 67.26 | Helped |
| add local-level distillation (ablation) | MMDist Student · Charades-STA | R@0.5: 51.68 -> 53.29 | Helped |
| add global-level distillation (ablation) | MMDist Student · Charades-STA | R@0.5: 52.85 -> 53.29 | Helped |
| add cross-modal mutual learning (ablation) | MMDist Teacher · Charades-STA | R@0.5: 53.86 -> 54.72 | Helped |
| replace the multi-modal teacher setup with teacher baseline without collaborative learning (ablation) | MMDist Teacher · Charades-STA | R@0.5: 54.72 -> 52.31 | Hurt |
Surprising findings
- The MMDist Student slightly exceeds its multi-modal Teacher on ActivityNet Captions at R@0.1 and R@0.3 (by about 0.2 points).
Limitations
- Teacher requires optical flow, which is computationally expensive to compute during training.
- Gains over the single-modal baseline are modest on some metrics, e.g., R@0.7 gains in ablations are small for the student.
Experimental setup
Datasets: Charades-STA ActivityNet Captions
Benchmarks: Charades-STA ActivityNet Captions
Metrics: R@0.3 R@0.5 R@0.7 R@0.1
Baselines: SCN BAR MARN RTBPN CCL WSTAN LCNet VCA CPL
Builds on: CPL (Gaussian-based contrastive proposal learning) knowledge distillation (Hinton et al.) I3D C3D GloVe
Closest papers by meaning
- SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · similarity 0.88)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
- Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Grounding (ICLR 2025 · similarity 0.87)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.87)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.86)
- Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.