Research map / Vision-language and multimodal / Cross-modal learning

Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Probabilistic embeddings for cross-modal retrieval · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes APSE-IPIK, which augments image-text training with BLIP-2-generated multi-perspective captions (inspire potential strategy) and selects image-conditioned prompts from a learnable prompt pool (instance-level query) on top of fine-tuned CLIP for cross-modal retrieval.

Problem

Cross-modal image-text retrieval suffers from information asymmetry between images and their text descriptions and from high intra-modal distribution diversity of images, which fixed prompts or plain encoder fine-tuning handle poorly.

Approach

Use a generative multimodal LLM (BLIP-2) with global, background, and entity questions to produce extra text descriptions that are aligned with images as augmented positives during training, and replace fixed prompts with a key-value prompt pool where each image retrieves its top-N prompts via cosine matching of CLIP query features.

Key results

• On Flickr30K, APSE-IPIK reaches RSUM 542.9 versus 521.1 for MGCN, with text-to-image R@1 of 72.0 vs 63.2.
• On MSCOCO 5K, APSE-IPIK reaches RSUM 439.8 versus 436.2 for MGCN, with R@10 of 94.6 for image-to-text retrieval.
• Ablation on Flickr30K: CLIP baseline RSUM 526.7, +inspire potential 532.1, +adaptive prompt pool 534.8, full model 542.9.

Measured effects reported in the paper

Surprising findings

• On MSCOCO, APSE-IPIK's image-to-text R@1 (59.1) is slightly below MGCN (59.3), and its RSUM gain over MGCN is small (about 3.6).

Limitations

• Generative caption model adds computational cost, which the authors plan to optimize.
• Hyperparameter results show performance sensitive to prompt pool size and selection count (M=20, N=5 chosen on test-reported RSUM).
• Small gain over strongest baseline on MSCOCO image-to-text R@1.

Experimental setup

Datasets: MSCOCO Flickr30K

Benchmarks: Flickr30K MSCOCO (5K)

Metrics: Recall@1 Recall@5 Recall@10 RSUM

Baselines: SCAN BFAN SGM IMRAM GSMN SMFEA SHAN VSE∞ SGRAF NAAF MGCN PVSE

Builds on: CLIP L2P BLIP-2 VSE∞

Closest papers by meaning

• Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.88)
• PromptHash:Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval (CVPR 2025 · similarity 0.88)
• Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval (AAAI 2025 · similarity 0.88)
• CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.88)
• Incorporating Dense Knowledge Alignment into Unified Multimodal Representation Models (CVPR 2025 · similarity 0.88)
• Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment (NeurIPS 2025 · similarity 0.88)