Research map / Video / Video-language understanding
Mitigating Semantic Collapse in Partially Relevant Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Addresses semantic collapse in partially relevant video retrieval by distilling CLIP's text relational structure (TCPL) and adding a cross-branch frame-clip contrastive alignment (CBVA) with order-preserving and bipartite token merging.
Problem
Partially relevant video retrieval (PRVR) trained with pairwise InfoNCE/triplet losses causes semantic collapse: queries and video segments from the same video are pulled together regardless of meaning, while semantically similar items from different videos are pushed apart.
Approach
Distill CLIP's pairwise Euclidean and triplet angular relations among text [EOS] tokens into the retrieval text space (TCPL), and align frame- and clip-level video tokens across branches with contrastive loss, using order-preserving and adaptive bipartite token merging to build coherent, distinct clip segments.
Key results
- On QVHighlights, the method reaches SumR 234.6 versus 225.2 for reproduced GMMFormer-v2 (R1 23.9 vs 21.7).
- On TVR with CLIP-L/14, SumR 263.1 vs 260.1 for ARL, the best prior reported result.
- Ablation on QVHighlights: TCPL raises SumR from 225.5 to 230.6; adaptive CBVA with OP-ToMe reaches 234.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TCPL to the retrieval baseline (ablation) | GMMFormer-v2 architecture on QVHighlights · QVHighlights | SumR: 225.5 -> 230.6 | Helped |
| add naïve CBVA after TCPL (ablation) | GMMFormer-v2 architecture on QVHighlights · QVHighlights | SumR: 230.6 -> 231.0 | Helped |
| add OP-ToMe to CBVA (ablation) | TCPL + CBVA on QVHighlights · QVHighlights | SumR: 231.0 -> 232.5 | Helped |
| adaptively adjust each video's clip count (ablation) | TCPL + OP-ToMe + CBVA on QVHighlights · QVHighlights | SumR: 232.5 -> 234.6 | Helped |
| use CBVA instead of retrieved-segment relation learning (ablation) | TCPL baseline on QVHighlights · QVHighlights | SumR: 231.7 -> 234.6 | Helped |
| use CLIP-L rather than CLIP-B as the TCPL source (ablation) | Ours on QVHighlights · QVHighlights | SumR: 234.6 -> 235.6 | Helped |
| use OpenCLIP-L rather than CLIP-B as the TCPL source (ablation) | Ours on QVHighlights · QVHighlights | SumR: 234.6 -> 236.4 | Helped |
| set TCPL loss ratio λE:λA to 1:2 instead of 1:1 (ablation) | Ours on QVHighlights · QVHighlights | SumR: 229.7 -> 234.6 | Helped |
| increase TCPL weights from (10, 20) to (15, 30) (ablation) | Ours on QVHighlights · QVHighlights | SumR: 233.5 -> 234.6 | Helped |
| increase TCPL weights from (15, 30) to (20, 40) (ablation) | Ours on QVHighlights · QVHighlights | SumR: 234.6 -> 232.5 | Hurt |
| set CBVA coefficient to 0.15 rather than 0.1 (ablation) | Ours on QVHighlights · QVHighlights | SumR: 234.6 -> 234.9 | Helped |
| lower token merge rate from 75 to 50 (ablation) | OP-ToMe and adaptive CBVA on QVHighlights · QVHighlights | SumR: 234.6 -> 232.6 | Hurt |
Surprising findings
- Naive cross-branch alignment with fixed-length clips gives only a marginal gain (SumR 230.6 to 231.0), and video-side CLIP distillation variants lag CBVA by 2.3 SumR.
- Relative to zero-shot CLIP, the method corrects more CLIP failures than it introduces on TVR (2551 vs 500 at R@1), but fails on some simple object-centric queries.
Limitations
- Inherits CLIP weaknesses, e.g., fine-grained spatial/directional queries such as left vs right.
- Increased training cost compared with prior PRVR methods.
- Gains on TVR, ActivityNet, and Charades are modest relative to QVHighlights.
Experimental setup
Datasets: QVHighlights TVR ActivityNet Captions Charades-STA
Benchmarks: QVHighlights TVR ActivityNet Captions Charades-STA
Metrics: R@1 R@5 R@10 R@100 SumR Spearman rank correlation
Baselines: MS-SL GMMFormer GMMFormer-v2 AMDNet BGM-Net ProtoPRVR ARL CLIP zero-shot DL-DKD
Builds on: GMMFormer-v2 CLIP ToMe Relational Knowledge Distillation MS-SL
Closest papers by meaning
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.92)
- Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.91)
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval (NeurIPS 2025 · similarity 0.89)
- Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval (AAAI 2025 · similarity 0.88)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.88)
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.