Research map / Video / Video-language understanding
Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GARE, a gap-aware learnable pair-specific increment module with a relaxed variational information bottleneck that offloads optimization tension in CLIP-based contrastive text-video retrieval.
Problem
Contrastive text-video retrieval suffers from optimization tension caused by the modality gap (anchor gradients from positives and negatives nearly cancel) and noisy gradients from false-negative hard negatives.
Approach
Introduce a learnable pair-specific increment Delta_ij, predicted by a cross-attention module conditioned on the semantic gap (v_j - t_i), added to the text anchor so gradients are redistributed from the shared anchor to pair-level increments; derive its update from a first-order Taylor expansion under a trust-region constraint and regularize with a relaxed variational information bottleneck plus
Key results
- On MSR-VTT 1k-A text-to-video retrieval, GARE reaches R@1 49.1 vs. 46.6 for the CLIP-based baseline (ablation with all losses).
- GARE reports R@1 48.6 / R@10 85.3 on MSR-VTT video-to-text retrieval, above the baseline's 45.6 / 82.4.
- On DiDeMo, GARE reaches R@1 47.6 vs. 45.4 for the baseline; on ActivityNet, R@1 42.6 vs. 40.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pair-specific increments Δ (ablation) | CLIP ViT-B/32 text-video retrieval · MSR-VTT 1k-A T2V | R@1: 46.6 to 47.4 | Helped |
| add relaxed information bottleneck loss to Δ (ablation) | GARE with pair-specific increments · MSR-VTT 1k-A T2V | R@1: 47.4 to 48.3 | Helped |
| add norm regularizer without IB (ablation) | GARE with pair-specific increments · MSR-VTT 1k-A T2V | R@1: 47.4 -> 47.2 | No clear effect |
| add directional diversity regularizer without IB (ablation) | GARE with pair-specific increments · MSR-VTT 1k-A T2V | R@1: 47.4 -> 47.0 | No clear effect |
| combine relaxed IB, norm, and directional diversity regularizers (ablation) | GARE · MSR-VTT 1k-A T2V | R@1: 48.3 to 49.1 | Helped |
| use video-frame rather than text-word context (ablation) | GARE · MSR-VTT 1k-A T2V | R@1: 47.4 -> 49.1 | Helped |
| use video-frame rather than text-word context (ablation) | GARE · ActivityNet Captions T2V | R@1: 42.6 -> 40.2 | Hurt |
| replace video-frame context with text-word context (ablation) | GARE · DiDeMo T2V | R@1: 47.6 -> 46.5 | Hurt |
| replace the semantic-gap query v_j - t_i with t_i (ablation) | GARE · MSR-VTT 1k-A T2V | R@1: 46.1 vs. 49.1 | Hurt |
| normalize Δ before KL regularization (ablation) | GARE · MSR-VTT 1k-A T2V | R@1: 47.8 vs. 49.1 | Hurt |
| use Gaussian prior σ=0.1 instead of σ=1 (ablation) | GARE with unnormalized Δ · MSR-VTT 1k-A T2V | R@1: 47.7 vs. 49.1 | Hurt |
| replace directional diversity with strict orthogonalization (ablation) | GARE · MSR-VTT 1k-A T2V | R@1: R@1=47.7, R@5=74.3, R@10=83.1, MnR=12.1 | Hurt |
Surprising findings
- Normalizing the increment before the KL regularizer accelerates convergence but lowers final accuracy, so the best setting keeps magnitude information.
- Adding the norm-diversity or directional-diversity regularizer alone yields no gain; they only help when combined with the relaxed IB loss.
Limitations
- The modality gap is alleviated rather than resolved; the two modalities still occupy disjoint regions of the embedding space.
- Increments are supervised only by the InfoNCE gradient, with no stronger generalizable supervision signal.
Experimental setup
Datasets: MSR-VTT DiDeMo MSVD ActivityNet Captions
Benchmarks: MSR-VTT DiDeMo ActivityNet Captions MSVD
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: CLIP4Clip TS2-Net X-Pool EMCL-Net UATVR DiCoSA DiffusionRet HBI
Builds on: InfoNCE CLIP4Clip variational information bottleneck
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.89)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.87)
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval (NeurIPS 2024 · similarity 0.87)
- VideoCon: Robust Video-Language Alignment via Contrast Captions (CVPR 2024 · similarity 0.87)
- Rethinking Noisy Video-Text Retrieval via Relation-aware Alignment (CVPR 2025 · similarity 0.87)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.