Research map / Video / Video-language understanding
Rethinking Noisy Video-Text Retrieval via Relation-aware Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Relation-aware Purified Consistency (RPC) network that mitigates noisy video-text correspondences using learned representative agents and a ranking-distribution alignment loss over agent permutations.
Problem
Video-text retrieval training assumes accurate video-text correspondences, but web-collected data contains noisy correspondences (mismatched pairs) that degrade retrieval performance, and existing noise-rectify methods are sensitive to clean-sample selection.
Approach
Builds reliable, representative, and resilient agent (reference pair) features without correspondence supervision, then aligns video and text agent-ranking distributions over permutations of top-4 agents as a relation-aware regularizer added to the InfoNCE loss.
Key results
- On MSR-VTT 1k-A at 50% noise, text-to-video R@1 reaches 44.6 versus 36.2 for the prior best RVTR.
- On MSR-VTT full at 50% noise, the method gains 8.7 R@1 over RVTR (text-to-video), and it degrades only 1.5 R@1 from 20% to 50% noise on 1k-A versus 6.1 for RVTR.
- On MSVD at 50% noise, text-to-video R@1 is 38.5 versus 28.7 for RVTR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RPC instead of RVTR | Video-text retrieval · MSR-VTT Full, 50% noise, text-to-video | R@1: 18.4 -> 27.1 | Helped |
| use RPC instead of RVTR | Video-text retrieval · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 36.2 -> 44.6 | Helped |
| use RPC instead of RVTR | Video-text retrieval · MSVD, 50% noise, text-to-video | R@1: 28.7 -> 38.5 | Helped |
| use RPC instead of RVTR | Video-text retrieval · LSMDC, 50% noise, text-to-video | R@1: 19.2 -> 22.8 | Helped |
| use RPC instead of RVTR | Video-text retrieval · DiDeMo, 50% noise, text-to-video | R@1: 26.9 -> 34.7 | Helped |
| use RPC instead of RVTR | Video-text retrieval · ActivityNet, 50% noise, text-to-video | R@1: 31.2 -> 33.0 | Helped |
| select agents by bidirectional correspondence rather than using all pairs, learnable agents, or random pairs (ablation) | RPC · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 43.1; 42.9; 42.8 -> 44.6 | Helped |
| add semantic purification mechanism (ablation) | RPC agent construction · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 40.4 -> 42.9 | Helped |
| add cross-aggregation (ablation) | RPC agent construction with semantic purification · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 42.9 -> 43.7 | Helped |
| add self-aggregation (ablation) | RPC agent construction with semantic purification and cross-aggregation · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 43.7 -> 44.6 | Helped |
| use relation-aware ranking-distribution alignment instead of pair-wise alignment (ablation) | RPC · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 39.3 -> 44.6 | Helped |
| use relation-aware ranking-distribution alignment instead of agent-level alignment (ablation) | RPC · MSR-VTT 1k-A, 50% noise, text-to-video | R@1: 42.6 -> 44.6 | Helped |
Surprising findings
- Pure pairwise similarity of noisy pairs can remain high (around 0.3) even when the pair is mismatched, while the rank consistency score separates them more clearly.
Limitations
- Evaluation of noise relies on synthetic caption shuffling rather than naturally noisy datasets.
- Computing all agent permutations is costly, so only top-4 agents are used, an approximation.
Experimental setup
Datasets: MSR-VTT LSMDC MSVD ActivityNet DiDeMo
Benchmarks: MSR-VTT LSMDC MSVD ActivityNet DiDeMo
Metrics: R@1 R@5 R@10 R@50
Baselines: CE MMT TT-CE+ T2VLAD CLIP4Clip RVTR
Builds on: CLIP4Clip NCR RVTR InfoNCE
Closest papers by meaning
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval (NeurIPS 2025 · similarity 0.87)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.86)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.86)
- TSVC: Tripartite Learning with Semantic Variation Consistency for Robust Image-Text Retrieval (AAAI 2025 · similarity 0.86)
- Multi-granularity Correspondence Learning from Long-term Noisy Videos (ICLR 2024 · similarity 0.85)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.