Research map / Vision-language and multimodal / Cross-modal learning
Statistical Model-driven Similarity Hashing: Bridging Modalities for Efficient Unsupervised Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMSH, an unsupervised deep cross-modal hashing method that adds Jaccard similarity to the text similarity matrix, fuses modality similarities into a unified matrix, and uses a GMM-based statistical similarity enhancement to push dissimilar image pairs apart.
Problem
Unsupervised deep cross-modal hashing suffers from insufficient modeling of text instance similarity (asymmetric binary BoW features), redundant fusion of multi-modal similarity, and noisy adjacent correlations that reduce hash code discriminability.
Approach
Combine cosine and Jaccard similarity for binary BoW text features, fuse image/text/cross-modal similarity matrices with balance weights, and apply a two-component Gaussian Mixture Model fitted to image similarities to down-weight low-similarity pairs via a sigmoid-like function.
Key results
- At 128 bits, SMSH reaches 0.865 I→T on NUS-WIDE, 0.942 I→T on MIRFlickr and 0.916 I→T on MSCOCO (mAP@50), beating the best deep baseline in each case per the paper's text.
- Ablation shows removing the SMSE step (SMSH-3) causes the largest drop, e.g. MSCOCO I→T at 128 bits falls from 0.916 to 0.891.
- GMM fit to image similarity has very small cross entropy (0.26036, 0.25003 and 0.48452 across datasets in Table 2), supporting the Gaussian modeling claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · NUS-WIDE I→T, 128 bits | mAP@50: 0.862 -> 0.865 | Helped |
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · NUS-WIDE T→I, 128 bits | mAP@50: 0.825 -> 0.837 | Helped |
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · MIRFlickr I→T, 128 bits | mAP@50: 0.937 -> 0.942 | Helped |
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · MIRFlickr T→I, 128 bits | mAP@50: 0.912 -> 0.917 | Helped |
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · MSCOCO I→T, 128 bits | mAP@50: 0.899 -> 0.916 | Helped |
| use SMSH instead of the strongest listed baseline | cross-modal retrieval · MSCOCO T→I, 128 bits | mAP@50: 0.919 -> 0.931 | Helped |
| add autoencoder reconstruction loss (ablation) | SMSH · NUS-WIDE, MIRFlickr and MSCOCO, I→T and T→I, 128 bits | mAP@50: 0.856 -> 0.865; 0.832 -> 0.837; 0.936 -> 0.942; 0.914 -> 0.917; 0.909 -> 0.916; 0.924 -> 0.931 | Helped |
| use Jaccard-plus-cosine text similarity instead of cosine-only (ablation) | SMSH · NUS-WIDE, MIRFlickr and MSCOCO, I→T and T→I, 128 bits | mAP@50: 0.855 -> 0.865; 0.829 -> 0.837; 0.939 -> 0.942; 0.914 -> 0.917; 0.909 -> 0.916; 0.928 -> 0.931 | Helped |
| add SMSE similarity enhancement (ablation) | SMSH · NUS-WIDE, MIRFlickr and MSCOCO, I→T and T→I, 128 bits | mAP@50: 0.839 -> 0.865; 0.826 -> 0.837; 0.930 -> 0.942; 0.913 -> 0.917; 0.891 -> 0.916; 0.908 -> 0.931 | Helped |
Surprising findings
- Text similarity is described as mainly concentrated near -1 because binary BoW vectors have many zeros, so the paper attributes the unified matrix's distribution mainly to image similarity.
Limitations
- Gains over the strongest baselines are small (often under 1 point mAP), and some settings are not best.
- Hyperparameters are tuned per dataset by cross-validation, and the choice of K=2 GMM components is fixed without analysis.
- The method relies on fixed AlexNet and BoW features rather than learned text encoders.
Experimental setup
Datasets: NUS-WIDE MIRFlickr MSCOCO
Benchmarks: NUS-WIDE MIRFlickr MSCOCO
Metrics: mAP@50 precision@N
Baselines: CVH IMH CMFH DBRC UDCMH DJSRH JDSH AGCH DAEH CIRH UCCH HUGH
Builds on: DJSRH JDSH CIRH DAEH
Closest papers by meaning
- Vision-guided Text Mining for Unsupervised Cross-modal Hashing with Community Similarity Quantization (AAAI 2025 · similarity 0.91)
- Robust Self-reflective Hashing for Cross-modal Retrieval with Noisy Label (ICML 2026 · similarity 0.90)
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.90)
- Effective Comparative Prototype Hashing for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.88)
- Asymmetric Cross-Modal Hashing Based on Formal Concept Analysis (AAAI 2025 · similarity 0.88)
- Deep Evidential Hashing for Trustworthy Cross-Modal Retrieval (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.