Research map / Vision-language and multimodal / Cross-modal learning
Test-time Adaptation for Cross-modal Retrieval with Query Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TCR, an online test-time adaptation method for cross-modal retrieval under query shift, combining a query prediction refinement module with a joint loss (intra-modality uniformity, inter-modality gap, noise-robust adaptation) and evaluated on COCO-C/Flickr-C and query-gallery shift benchmarks.
Problem
Cross-modal retrieval models pre-trained on a source domain degrade when online queries follow a different distribution (query shift), and existing test-time adaptation methods designed for recognition are noisy and ill-suited to retrieval with large gallery sets.
Approach
Observes that query shift reduces query-modality uniformity and widens the modality gap; refines query predictions by restricting the softmax to each query's nearest gallery candidate, selects source-like pairs to estimate the source modality gap and an entropy threshold, and optimizes a joint loss that enlarges uniformity, matches the estimated gap, and down-weights noisy predictions.
Key results
- On COCO-C with maximum severity, TCR reaches 59.1 average R@1 on image-modality shift for BLIP ViT-B/16 versus 46.0 for the best baseline (EATA-like baselines, 49.3 for EATA).
- On Base2COCO with BLIP ViT-B/16, TCR attains TR@1 68.9 and IR@1 48.9, versus 64.2/47.9 for EATA.
- On ReID benchmarks with CLIP ViT-B/16, TCR improves CUHK2ICFG IR@1 from 33.3 to 37.3 over baselines around 33.0-33.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply TCR rather than EATA | BLIP ViT-B/16 · COCO-C image query shift, max severity | Recall@1 Avg.: 49.3 -> 59.1 | Helped |
| apply TCR rather than EATA | BLIP ViT-L/16 · COCO-C text query shift, max severity | Recall@1 Avg.: 43.7 -> 44.4 | Helped |
| apply TCR rather than DeYO | BLIP ViT-B/16 · Query-Gallery shifts, benchmark average | Recall@1 Avg.: 64.2 -> 67.0 | Helped |
| apply TCR rather than EATA | CLIP ViT-B/16 · CUHK2ICFG and ICFG2CUHK | IR@1 Avg.: 37.8 -> 39.9 | Helped |
| add noise-robust adaptation loss (ablation) | TCR with intra-modality uniformity and inter-modality gap losses · Base2COCO | TR@1: 66.3 -> 68.9 | Helped |
| add intra-modality uniformity loss (ablation) | TCR with noise-robust adaptation and inter-modality gap losses · Base2COCO | IR@1: 48.4 -> 48.9 | Helped |
| add inter-modality gap loss (ablation) | TCR with noise-robust adaptation and intra-modality uniformity losses · Base2COCO | TR@1: 67.8 -> 68.9 | Helped |
| increase query-modality intra-modality uniformity | cross-modal retrieval embeddings · Base2COCO analytic study | retrieval performance | Helped |
| monotonously eliminate the query-gallery modality gap | cross-modal retrieval embeddings · Base2COCO analytic study | retrieval performance | Mixed |
| use untrained TCR rather than EATA | BLIP ViT-B/16 · COCO-C image query shift, max severity | Recall@1 Avg.: 49.3 -> 52.7 | Helped |
Surprising findings
- Baseline TTA methods often give only marginal gains or even degrade performance as the gallery grows (e.g., Base2Fashion), indicating they are ill-suited to retrieval.
- An untrained variant of TCR (scaling queries and matching the source modality gap, no parameter update) still outperforms EATA on COCO-C and Flickr-C.
Limitations
- Evaluation limited to image-text retrieval and ReID with CLIP and BLIP backbones; generalization to other modalities is not shown.
- Hyperparameters such as the temperature must be tuned per dataset for baselines, and the method relies on a source-like pair selection heuristic (30% smallest SI).
Experimental setup
Datasets: COCO Flickr30k Fashion-Gen CUHK-PEDES ICFG-PEDES NoCaps
Benchmarks: COCO-C Flickr-C Fashion-Gen CUHK-PEDES ICFG-PEDES NoCaps Flickr30k COCO
Metrics: Recall@1
Baselines: Tent EATA SAR READ DeYO
Builds on: Tent EATA SAR DeYO READ
Closest papers by meaning
- Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data (ICML 2025 · similarity 0.85)
- Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024 · similarity 0.85)
- NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval (CVPR 2025 · similarity 0.84)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.84)
- DiCA: Disambiguated Contrastive Alignment for Cross-Modal Retrieval with Partial Labels (AAAI 2025 · similarity 0.84)
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.