Research map / Vision-language and multimodal / Cross-modal learning
Vision-guided Text Mining for Unsupervised Cross-modal Hashing with Community Similarity Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VTM-UCH, an unsupervised cross-modal hashing method that uses CLIP word-to-vision matching, K-means vision clustering for text similarity, and Leiden community detection to refine pairwise similarity targets.
Problem
Unsupervised cross-modal hashing suffers from weak and noisy text modality, and pairwise similarity losses yield poorly aggregated hash code distributions.
Approach
Match each word to its most similar image or detected object via CLIP, derive text similarity from shared K-means vision clusters, fuse with fine-grained object-level image similarity, and use Leiden community detection as pseudo-centers to refine the joint similarity matrix for a pairwise hashing loss.
Key results
- On MIRFLICKR-25K I→T at 16 bits, VTM-UCH reaches 0.743 mAP versus 0.731 for the strongest baseline UCHSTM.
- On MIRFLICKR-25K T→I at 16 bits, VTM-UCH reaches 0.751 mAP versus 0.733 for UCHSTM.
- On NUS-WIDE the method does not uniformly win; e.g., I→T at 16 bits it gets 0.616 versus UCHSTM's 0.623.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VTM-UCH instead of UCHSTM | cross-modal hashing · MIRFLICKR-25K, I → T, 128 bits | mAP: 0.751 -> 0.754 | Helped |
| Use VTM-UCH instead of CIRH | cross-modal hashing · MIRFLICKR-25K, T → I, 128 bits | mAP: 0.734 -> 0.761 | Helped |
| Use VTM-UCH instead of UCHSTM | cross-modal hashing · NUS-WIDE, I → T, 128 bits | mAP: 0.649 -> 0.631 | Hurt |
| Use VTM-UCH instead of UCHSTM | cross-modal hashing · NUS-WIDE, T → I, 128 bits | mAP: 0.658 -> 0.645 | Hurt |
| Add community detection to the fused similarity matrix (ablation) | VTM-UCH · MIRFLICKR-25K, I → T, 32 bits | mAP: 0.752 -> 0.754 | No clear effect |
| Add community detection to the fused similarity matrix (ablation) | VTM-UCH · MIRFLICKR-25K, T → I, 32 bits | mAP: 0.756 -> 0.758 | Helped |
| Use the community-label similarity matrix directly in the loss (ablation) | VTM-UCH · MIRFLICKR-25K, I → T, 32 bits | mAP: 0.721 -> 0.754 | Hurt |
| Use the community-label similarity matrix directly in the loss (ablation) | VTM-UCH · MIRFLICKR-25K, T → I, 32 bits | mAP: 0.723 -> 0.758 | Hurt |
| Match words using only original images rather than images and detected objects (ablation) | VTM-UCH · MIRFLICKR-25K, I → T, 32 bits | mAP: 0.733 -> 0.754 | Hurt |
| Match words using only original images rather than images and detected objects (ablation) | VTM-UCH · MIRFLICKR-25K, T → I, 32 bits | mAP: 0.736 -> 0.758 | Hurt |
| Use FasterR-CNN trained on MS-COCO for object detection (ablation) | VTM-UCH · MIRFLICKR-25K, I → T, 32 bits | mAP: 0.731 -> 0.754 | Hurt |
| Use FasterR-CNN trained on MS-COCO for object detection (ablation) | VTM-UCH · MIRFLICKR-25K, T → I, 32 bits | mAP: 0.732 -> 0.758 | Hurt |
Surprising findings
- On NUS-WIDE, VTM-UCH does not consistently outperform UCHSTM, with several settings slightly lower.
Limitations
- Gains are mixed on NUS-WIDE, where some baselines match or exceed the method.
- Image similarity computation is O(N^2) and is approximated by limiting objects per image, which becomes expensive for large training sets.
- Relies on hyperparameters such as cluster count Nc and detection threshold tau that are dataset-specific.
Experimental setup
Datasets: MIRFLICKR-25K NUS-WIDE Open Images Dataset MS-COCO
Benchmarks: MIRFLICKR-25K NUS-WIDE
Metrics: mAP precision-recall curves precision recall
Baselines: CVH IMH CMFH FSH RFDH DJSRH JDSH DSAH UKD-SS DAEH CIRH UCHSTM
Builds on: CLIP Leiden algorithm UCHSTM Faster R-CNN
Closest papers by meaning
- Statistical Model-driven Similarity Hashing: Bridging Modalities for Efficient Unsupervised Retrieval (AAAI 2025 · similarity 0.91)
- Collaborative Quantization for Cross-Modal Similarity Search (CVPR 2016 · similarity 0.91)
- Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval (AAAI 2025 · similarity 0.91)
- UGGRH: Unsupervised Generative Completion and Graph-attention Refinement for Incomplete Cross-modal Hashing (NeurIPS 2026 · similarity 0.90)
- UDCH: Unsupervised Dynamic Weighted Cluster-cooperative Hashing for Cross-modal Retreival (AAAI 2026 · similarity 0.90)
- An End-To-End Graph Attention Network Hashing for Cross-Modal Retrieval (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.