Research map / Vision-language and multimodal / Cross-modal learning
NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeighborRetr, a training-time method that mitigates cross-modal hubness via centrality weighting, neighbor adjusting, and uniform regularization on CLIP-based text-video and text-image retrieval.
Problem
Cross-modal retrieval suffers from hubness, where a few hub samples dominate nearest-neighbor lists, biasing representations and degrading retrieval; existing fixes are post-hoc and need prior data distributions.
Approach
Measure intra-modal sample centrality with a memory bank, weight the contrastive loss by centrality, de-centre the cross-modal similarity (S minus gallery centrality) to adaptively separate good from bad neighbors, and add an optimal-transport-style uniform marginal regularizer to lift anti-hubs.
Key results
- On MSR-VTT text-to-video, NeighborRetr reaches R@1 49.5 and Rsum 207.7, and reduces hubness skew by 46% versus the vanilla-loss baseline (3.20 vs 5.97).
- On ECCV Caption text-to-image, R@1 reaches 92.1 vs 90.8 for PCME++.
- On ActivityNet Captions text-to-video, R@1 improves to 46.0 from 42.2 (HBI), a 3.8-point gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeighborRetr instead of Diffusion | CLIP ViT-B/32 text-video retrieval · MSR-VTT text-to-video | R@1: 49.0 -> 49.5 | Helped |
| use NeighborRetr instead of Diffusion | CLIP ViT-B/32 text-video retrieval · MSR-VTT video-to-text | R@1: 47.7 -> 48.7 | Helped |
| use NeighborRetr instead of UATVR | CLIP ViT-B/32 text-video retrieval · MSVD text-to-video | R@1: 46.0 -> 47.9 | Helped |
| use NeighborRetr instead of HBI | CLIP ViT-B/32 text-video retrieval · ActivityNet Captions text-to-video | R@1: 42.2 -> 46.0 | Helped |
| use NeighborRetr instead of HBI | CLIP ViT-B/32 text-video retrieval · DiDeMo text-to-video | R@1: 46.9 -> 48.2 | Helped |
| use NeighborRetr instead of PCME++ | CLIP ViT-B/32 cross-modal retrieval · ECCV Caption text-to-image | R@1: 90.8 -> 92.1 | Helped |
| use NeighborRetr instead of PCME++ | CLIP ViT-B/32 cross-modal retrieval · ECCV Caption image-to-text | R@1: 81.3 -> 82.1 | Helped |
| add LOpt to the LWti-only model (ablation) | NeighborRetr · MSR-VTT text-to-video | R@sum: +0.8% | Helped |
| add LNbi to the LWti-only model (ablation) | NeighborRetr · MSR-VTT text-to-video | R@sum: +2.0% | Helped |
| add LOpt, LNbi, and LKL to LWti (ablation) | NeighborRetr · MSR-VTT text-to-video | R@sum: 204.7 -> 207.7 | Helped |
| use intra-modal centrality for LWti and inter-modal centrality for LNbi (ablation) | NeighborRetr · MSR-VTT text-to-video | R@sum: 204.1 -> 207.7 | Helped |
| replace Simi-Cent with Simi-Cent's individual similarity term (Simi) (ablation) | NeighborRetr neighborhood loss · MSR-VTT text-to-video | R@sum: 207.7 -> 206.5 | Hurt |
Surprising findings
- Uniform-marginal regularization and the neighbor-adjusting loss each improve retrieval, but adding both with centrality weighting is needed for best results; additive similarity+centrality combination gives only marginal gains over the vanilla baseline.
Limitations
- Relies on a memory bank (size 10,240) and neighbor-size hyperparameter (optimal 20) that must be tuned.
- Evaluated mainly with CLIP ViT-B/32 backbone and fine-grained two-level token-merging framework.
Experimental setup
Datasets: MSR-VTT MSVD ActivityNet Captions DiDeMo MS-COCO ECCV Caption
Benchmarks: MSR-VTT MSVD ActivityNet Captions DiDeMo ECCV Caption
Metrics: R@1 R@5 R@10 Rsum MdR MnR mAP@R R-Precision skew trunc
Baselines: CLIP4Clip X-Pool TS2-Net EMCL-Net DiCoSA HBI DiffusionRet MPT UATVR PCME++ CUSA QB-Norm
Builds on: CLIP4Clip HBI HAL QB-Norm DPC-KNN
Closest papers by meaning
- Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval (AAAI 2026 · similarity 0.89)
- Intra-Modal Neighbors Never Lie: Rectifying Inter-Modal Noisy Correspondence via Graph-Based Intra-Modal Reasoning (ICML 2026 · similarity 0.88)
- A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval (NeurIPS 2022 · similarity 0.88)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.87)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.