Research map / Large language models / Language, safety and interpretability
Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniD, a weakly-supervised natural language video localization method that trains a teacher ensemble of CPL sub-models with intra-model IoU-consensus pseudo-labels and distills it into a student using GPT-3.5 paraphrased queries, plus six new language-variant test sets.
Problem
Weakly-supervised natural language video localization suffers from insufficient supervision (only video-level pairs) and limited query writing styles in benchmarks like Charades-STA and ActivityNet-Captions, hurting generalization to diverse real-world queries.
Approach
Bootstraps a teacher from N sub-models that pseudo-label each other via consensus IoU, then uses LLM paraphrases of queries to enforce semantic equivariance: the student's localization on a paraphrased query should match the teacher's on the original query.
Key results
- On Charades-STA SS, OmniD with LLM reaches R@0.5 of 41.41 vs 38.78 for CPL without LLM (paraphrasing-enhanced CPL reported in table).
- On vanilla Charades-STA, the OmniD student improves R@0.7 by more than 8 points over CPL (24.35 vs 22.39).
- On the ActivityNet-Captions VO variant with LLM, OmniD surpasses CPL by over 3 points in R@0.5 (48.55 vs 47.87 R@0.3 comparison; R@0.5 28.83 vs 27.13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-paraphrased queries with semantic equivariance distillation (ablation) | OmniD · Charades-STA SS | R@0.5: 39.98 -> 49.09 | Helped |
| Add semantic equivariance distillation while using LLM paraphrasing (ablation) | OmniD · Charades-STA SS | R@0.5: 44.65 -> 49.09 | Helped |
| Add LLM paraphrasing while using semantic equivariance distillation (ablation) | OmniD · Charades-STA SS | R@0.5: 41.41 -> 49.09 | Helped |
| Remove intra-model consistency distillation (ablation) | OmniD teacher · Charades-STA | R@0.5: 53.77 -> 51.68 | Hurt |
| Replace bootstrapping with a single sub-model (ablation) | OmniD teacher · Charades-STA | R@0.5: 53.77 -> 50.44 | Hurt |
| Use OmniD instead of CPL with LLM paraphrasing | Charades-STA SS · Charades-STA SS | R@0.5: 44.74 -> 49.09 | Helped |
| Use OmniD instead of CPL with LLM paraphrasing | Charades-STA VO · Charades-STA VO | R@0.5: 42.95 -> 48.63 | Helped |
| Use OmniD instead of CPL with LLM paraphrasing | Charades-STA HY · Charades-STA HY | R@0.5: 43.96 -> 49.04 | Helped |
| Use OmniD instead of CPL with LLM paraphrasing | ActivityNet-Captions SS · ActivityNet-Captions SS | R@0.5: 26.47 -> 28.50 | Helped |
| Use OmniD instead of CPL with LLM paraphrasing | ActivityNet-Captions VO · ActivityNet-Captions VO | R@0.5: 26.52 -> 29.53 | Helped |
| Use OmniD instead of CPL with LLM paraphrasing | ActivityNet-Captions HY · ActivityNet-Captions HY | R@0.5: 26.57 -> 28.99 | Helped |
Surprising findings
- LLM paraphrasing gives only modest, sometimes unstable gains for prior methods (e.g., SCN gains about one point on Charades-STA HY), while OmniD gains consistently.
Limitations
- Relies on manual checking of semantic equivariance for the variant test sets.
- Gains over baselines on variant datasets are sometimes modest and the approach depends on a proprietary LLM (GPT-3.5).
Experimental setup
Datasets: Charades-STA ActivityNet-Captions
Benchmarks: Charades-STA ActivityNet-Captions Charades-STA SS Charades-STA VO Charades-STA HY ActivityNet-Captions SS ActivityNet-Captions VO ActivityNet-Captions HY
Metrics: R@0.3 R@0.5 R@0.7 R@0.1
Baselines: SCN CPL BAR MARN RTBPN CCL LCNet VCA WSTAN
Builds on: CPL SCN Hinton knowledge distillation
Closest papers by meaning
- Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization (AAAI 2025 · similarity 0.88)
- Commonsense for Zero-Shot Natural Language Video Localization (AAAI 2024 · similarity 0.88)
- Unsupervised Domain Adaptative Temporal Sentence Localization with Mutual Information Maximization (AAAI 2024 · similarity 0.87)
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- OmniViD: A Generative Framework for Universal Video Understanding (CVPR 2024 · similarity 0.87)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.