Research map / Vision-language and multimodal / Cross-modal learning
PromptHash:Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptHash, an end-to-end CLIP-based cross-modal hashing framework with text affinity prompts, an SSM-Transformer adaptive gated fusion module, and a prompt affinity contrastive loss, reporting large mAP gains on MIRFLICKR-25K, NUS-WIDE and MS COCO.
Problem
Cross-modal hashing with CLIP suffers from a 77-token context limit that truncates long texts, plus redundant or background information in image-text features and modality heterogeneity that hurt retrieval accuracy.
Approach
Generate label-based text prompts with learnable context, weight them against the original text via a Transformer and learnable weights, fuse image and prompt-text features with a Mamba-style SSM with flip-based scanning plus gated selection, and align them with global and local (temperature-adjusted) InfoNCE plus affinity losses.
Key results
- On NUS-WIDE, PromptHash improves mAP@all over VTPH by 18.22% (I2T) and 18.65% (T2I), per the paper's text.
- On MS COCO, PromptHash exceeds the previous best (CMCL) by 8.87% (I2T) and 8.98% (T2I) mAP.
- On MIRFLICKR-25K, the 64-bit I2T mAP reaches 0.9995 (average 0.9924 across bit lengths), versus 0.9328 for VTPH at 64 bits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PromptHash instead of VTPH | Cross-modal hashing retrieval · MIRFLICKR-25K I2T | mAP@ALL: 7.13% | Helped |
| Use PromptHash instead of VTPH | Cross-modal hashing retrieval · MIRFLICKR-25K T2I | mAP@ALL: 7.47% | Helped |
| Use PromptHash instead of VTPH | Cross-modal hashing retrieval · NUS-WIDE I2T | mAP@ALL: 18.22% | Helped |
| Use PromptHash instead of VTPH | Cross-modal hashing retrieval · NUS-WIDE T2I | mAP@ALL: 18.65% | Helped |
| Use PromptHash instead of CMCL | Cross-modal hashing retrieval · MS COCO I2T | mAP@ALL: 8.87% | Helped |
| Use PromptHash instead of CMCL | Cross-modal hashing retrieval · MS COCO T2I | mAP@ALL: 8.98% | Helped |
| Add TAAP, AGSF, and PACL to the CLIP-feature-and-hash baseline (ablation) | PromptHash · MIRFLICKR-25K, NUS-WIDE, and MS COCO I2T, 32bit | mAP: 0.8590 -> 0.9960; 0.7377 -> 0.9759; 0.7594 -> 0.8782 | Helped |
| Add AGSF and PACL while retaining TAAP (ablation) | PromptHash · MIRFLICKR-25K, NUS-WIDE, and MS COCO I2T, 32bit | mAP: 0.8855 -> 0.9960; 0.8555 -> 0.9759; 0.7910 -> 0.8782 | Helped |
| Add TAAP and PACL while retaining AGSF (ablation) | PromptHash · MIRFLICKR-25K, NUS-WIDE, and MS COCO I2T, 32bit | mAP: 0.9269 -> 0.9960; 0.7838 -> 0.9759; 0.7687 -> 0.8782 | Helped |
| Add AGSF while retaining TAAP and PACL (ablation) | PromptHash · MIRFLICKR-25K, NUS-WIDE, and MS COCO I2T, 32bit | mAP: 0.9512 -> 0.9960; 0.8584 -> 0.9759; 0.8111 -> 0.8782 | Helped |
| Add PACL while retaining TAAP and AGSF (ablation) | PromptHash · MIRFLICKR-25K, NUS-WIDE, and MS COCO I2T, 32bit | mAP: 0.9859 -> 0.9960; 0.9450 -> 0.9759; 0.8228 -> 0.8782 | Helped |
Surprising findings
- Gains are much smaller on MS COCO, which the authors attribute to long full-sentence captions with only 3-5 retrieval-relevant words.
Limitations
- Modest improvements on MS COCO due to noisy long captions, which the authors leave to future work.
- Many hyperparameters (six loss weights) tuned per dataset via sweeps.
Experimental setup
Datasets: MIRFLICKR-25K NUS-WIDE MS COCO
Benchmarks: MIRFLICKR-25K NUS-WIDE MS COCO
Metrics: mAP@all precision-recall curves
Baselines: DCMH CMHH GCDH DCHMT MITH DSPH TwDH DNpH DHaPH CMCL VTPH
Builds on: CLIP CoCoOp VTPH DCMH
Closest papers by meaning
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.89)
- Dual Self-Paced Cross-Modal Hashing (AAAI 2024 · similarity 0.88)
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval (AAAI 2025 · similarity 0.88)
- Exploiting Descriptive Completeness Prior for Cross Modal Hashing with Incomplete Labels (NeurIPS 2024 · similarity 0.88)
- An End-To-End Graph Attention Network Hashing for Cross-Modal Retrieval (NeurIPS 2024 · similarity 0.88)
- Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.