Research map / Vision-language and multimodal / Cross-modal learning
DiCA: Disambiguated Contrastive Alignment for Cross-Modal Retrieval with Partial Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiCA, a method for cross-modal retrieval trained only with partial (candidate) label sets, combining a non-candidate boosted disambiguation loss with instance- and prototype-level cross-modal alignment.
Problem
Cross-modal retrieval when each image-text pair is annotated only with a set of candidate labels containing the true label, which creates label ambiguity and a modality gap.
Approach
Non-candidate boosted disambiguation learning (NBDL) weights losses on candidate and non-candidate labels and projects both modalities into a common space; instance-prototype representation learning (IPRL) adds pseudo-label-guided cross-modal contrast (ICC) and prototype-wise MAE alignment (PCA) to close the modality gap.
Key results
- On Wikipedia at partial rate 0.1, DiCA reaches 0.505 mAP for image-to-text retrieval, above the best baseline values in Table 1.
- At partial rate 0.4 on Wikipedia, DiCA exceeds SCL and PaPi by 8.3% and 11.5% for image-to-text retrieval, per the authors.
- On INRIA-Websearch at partial rate 0.01, DiCA achieves 0.565 mAP for image-to-text retrieval versus 0.544 for PaPi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiCA instead of the strongest listed baseline | Cross-modal retrieval with partial labels · Wikipedia, partial rate 0.4, Image → Text | mAP: 0.376 -> 0.469 | Helped |
| Use DiCA instead of the strongest listed baseline | Cross-modal retrieval with partial labels · NUS-WIDE, partial rate 0.4, Image → Text | mAP: 0.676 -> 0.682 | Helped |
| Use DiCA instead of the strongest listed baseline | Cross-modal retrieval with partial labels · INRIA-Websearch, partial rate 0.04, Text → Image | mAP: 0.479 -> 0.560 | Helped |
| Use DiCA instead of the strongest listed baseline | Cross-modal retrieval with partial labels · XMediaNet, partial rate 0.04, Image → Text | mAP: 0.391 -> 0.427 | Helped |
| Add ICC loss to NBDL and PCA (ablation) | DiCA · Wikipedia, partial rate 0.4, Image → Text | mAP: 0.430 -> 0.469 | Helped |
| Add PCA loss to NBDL and ICC (ablation) | DiCA · Wikipedia, partial rate 0.4, Image → Text | mAP: 0.389 -> 0.469 | Helped |
| Add ICC loss to NBDL and PCA (ablation) | DiCA · INRIA-Websearch, partial rate 0.04, Image → Text | mAP: 0.527 -> 0.542 | Helped |
| Add PCA loss to NBDL and ICC (ablation) | DiCA · INRIA-Websearch, partial rate 0.04, Text → Image | mAP: 0.523 -> 0.560 | Helped |
Surprising findings
- Unsupervised cross-modal methods (e.g., SCL) remain competitive with partial-label methods on some settings, and partial-label methods' accuracy drops fast as partial rate increases.
Limitations
- Performance of partial-label baselines degrades significantly with higher partial rates.
- Evaluated only on four benchmarks with simple pretrained feature extractors (VGG-19, Doc2Vec, AlexNet, LDA).
Experimental setup
Datasets: Wikipedia NUS-WIDE INRIA-Websearch XMediaNet
Benchmarks: Wikipedia NUS-WIDE INRIA-Websearch XMediaNet
Metrics: mAP
Baselines: MCCA PLS DCCA DCCAE SCL CC RC PRODEN LWS PaPi SCARCE
Builds on: LW (leveraged weighted loss) PiCO DCCA
Closest papers by meaning
- Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (AAAI 2024 · similarity 0.89)
- Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.87)
- Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels (NeurIPS 2024 · similarity 0.87)
- DiDA: Disambiguated Domain Alignment for Cross-Domain Retrieval with Partial Labels (AAAI 2024 · similarity 0.87)
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.