Research map / Video / Video-language understanding
Multi-granularity Correspondence Learning from Long-term Noisy Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Norton, a noise-robust temporal optimal transport framework that handles coarse- and fine-grained noisy correspondence for long-video-paragraph and clip-caption learning on HowTo100M-pretrained VideoCLIP-initialized dual encoders.
Problem
Learning long-term temporal correspondence between video clips and captions is hampered by multi-granularity noisy correspondence: clip-caption misalignment (asynchronous/irrelevant) and frame-word misalignment, plus faulty negatives in contrastive learning.
Approach
Unifies video-paragraph and clip-caption contrastive learning under an entropic optimal transport framework, adding a log-sum-exp soft-maximum for fine-grained frame-word similarity, an alignable prompt bucket to discard irrelevant clips/captions, and OT-based soft targets to correct faulty negatives in clip-caption contrast.
Key results
- On YouCookII video-paragraph retrieval (background removed, DTW), Norton reaches R@1 88.7 vs TempCLR 83.5.
- On COIN action segmentation, Norton achieves 69.8 frame accuracy vs 68.7 for VideoCLIP and TempCLR.
- Zero-shot MSR-VTT VideoQA accuracy improves from 74.4 (TempCLR) to 77.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Norton instead of TempCLR for video-paragraph retrieval with DTW | VideoCLIP · YouCookII, background removed | R@1: 83.5 -> 88.7 | Helped |
| use Norton instead of TempCLR for video-paragraph retrieval with DTW | VideoCLIP · YouCookII, background kept | R@1: 70.4 -> 76.1 | Helped |
| use Norton instead of TempCLR for clip-caption retrieval | VideoCLIP · YouCookII | R@1: 23.3 -> 24.2 | Helped |
| use Norton instead of VideoCLIP | VideoCLIP · MSR-VTT text-to-video retrieval, zero-shot | R@5: 22.2 -> 24.1 | Helped |
| use Norton instead of TempCLR | VideoCLIP · MSR-VTT VideoQA, zero-shot | accuracy: 74.4 -> 77.1 | Helped |
| use Norton instead of VideoCLIP | VideoCLIP · COIN action segmentation | frame accuracy: 68.7 -> 69.8 | Helped |
| remove video-paragraph contrastive loss (ablation) | Norton · YouCookII clip-caption retrieval | R@1: 24.2 -> 22.8 | Hurt |
| add faulty negative exploitation (ablation) | Norton without video-paragraph contrastive loss · YouCookII clip-caption retrieval | R@1: 22.8 -> 23.4 | Helped |
| add faulty negative exploitation (ablation) | Norton without video-paragraph contrastive loss · YouCookII video-paragraph retrieval, background removed, DTW | R@1: 56.7 -> 63.3 | Helped |
| replace mean average pooling with fine-grained soft-max similarity (ablation) | Norton · YouCookII video-paragraph retrieval, background removed, DTW | R@1: 84.2 -> 88.4 | Helped |
| use log-sum-exp soft-max instead of most-important-token alignment (ablation) | Norton · YouCookII video-paragraph retrieval, background removed, DTW | R@1: 86.9 -> 88.4 | Helped |
| increase soft-max α from 0.1 to 1 (ablation) | Norton · YouCookII video-paragraph retrieval, background removed, DTW | R@1: 88.1 -> 88.4 | Helped |
Limitations
- Only bi-modal video-text alignment; extension to multi-modal noisy correspondence is left open due to combinatorial growth.
- Gains on clip-caption and some benchmarks are small (e.g., ~1% on Cap. Avg. R@1), and hyperparameters like prompt value p and alpha are tuned on the evaluation setup.
Experimental setup
Datasets: HowTo100M YouCookII MSR-VTT COIN HTM-Align
Benchmarks: YouCookII MSR-VTT COIN HTM-Align
Metrics: R@1 R@5 R@10 frame-wise accuracy accuracy recall
Baselines: VideoCLIP TempCLR MIL-NCE TAN MCN HT100M OTAM ClipBERT
Builds on: VideoCLIP TempCLR Dynamic Time Warping Sinkhorn-Knopp
Closest papers by meaning
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models (CVPR 2025 · similarity 0.87)
- VERIFIED: A Video Corpus Moment Retrieval Benchmark for Fine-Grained Video Understanding (NeurIPS 2024 · similarity 0.86)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.86)
- Rethinking Noisy Video-Text Retrieval via Relation-aware Alignment (CVPR 2025 · similarity 0.85)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.85)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.