Research map / Vision-language and multimodal / Cross-modal learning
Multi-Level Cross-Modal Alignment for Image Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-level Cross-modal Alignment (MCA), which filters WordNet nouns into a smaller semantic space and aligns frozen CLIP image and text embeddings at instance, prototype, and semantic levels to improve deep image clustering.
Problem
Image clustering with cross-modal pretrained models like CLIP produces erroneous image-text alignments that yield poor pseudo-labels and degraded clustering performance.
Approach
Builds a smaller WordNet-based semantic space via uniqueness-based and hierarchy-based filtering, then aligns images and texts at three levels (instance-level contrastive, prototype-level contrastive, and attention-weighted semantic-level pseudo-label alignment) while training cluster heads on top of frozen CLIP encoders.
Key results
- Best-run ACC of 98.2 on STL10, 92.8 on Cifar10, 61.2 on Cifar100-20, 77.9 on ImageNet-Dogs, and 61.9 on Tiny-ImageNet, outperforming SIC (Best) on all five datasets.
- On ImageNet-Dogs, the full three-level alignment with L_I reaches 74.8 ACC, versus 46.7 with only image consistency loss.
- Reduces the number of WordNet words by up to 60% relative to SIC while improving clustering performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MCA instead of SIC (best) | CLIP image clustering · ImageNet-Dogs | ACC: 71.3 -> 77.9 | Helped |
| Use MCA instead of SIC (best) | CLIP image clustering · Cifar100-20 | ACC: 58.4 -> 61.2 | Helped |
| Add hierarchy-based filtering after uniqueness-based filtering (ablation) | semantic space construction · ImageNet-Dogs | ACC: 65.3 -> 71.8 | Helped |
| Add hierarchy-based filtering after uniqueness-based filtering (ablation) | semantic space construction · Cifar10 | ACC: 91.4 -> 91.9 | Helped |
| Add instance-level alignment loss (ablation) | MCA · ImageNet-Dogs | ACC: 53.1 {$\pm$3.8} -> 74.8 {$\pm$2.7} | Helped |
| Add prototype-level alignment loss (ablation) | MCA · ImageNet-Dogs | ACC: 63.5 {$\pm$2.5} -> 74.8 {$\pm$2.7} | Helped |
| Add semantic-level alignment loss (ablation) | MCA · ImageNet-Dogs | ACC: 66.0 {$\pm$3.4} -> 74.8 {$\pm$2.7} | Helped |
| Add all three cross-modal alignment strategies (ablation) | image consistency learning · ImageNet-Dogs | ACC: 46.7{$\pm$0.5} -> 74.8 {$\pm$2.7} | Helped |
| Increase the number of neighboring texts k_S too much (ablation) | MCA cross-modal alignment · Image clustering | clustering performance | Hurt |
| Vary the number of prototype neighbors k_p (ablation) | MCA prototype-level alignment · Image clustering | clustering performance | No clear effect |
| Vary trade-off parameters η, λ_a, λ_pa, and λ_sa (ablation) | MCA · STL10 | clustering performance | No clear effect |
| Increase η, λ_pa, and λ_sa (ablation) | MCA · ImageNet-Dogs | clustering performance | Helped |
Surprising findings
- Semantic-level alignment yields the largest single gain in ablation, and the method is reported as trailing MUST on some datasets because it lacks class names.
- The method is slower than SIC (e.g., 1 vs 0.5 hours on STL10 and 5 vs 4 hours on Tiny-ImageNet in training time) and is not as cost-effective as SIC.
Limitations
- Training three networks makes the method more costly than SIC.
- Performance lags MUST because class names are not used.
- Setting of hierarchy levels remains a challenge needing further investigation.
Experimental setup
Datasets: STL10 Cifar10 Cifar100-20 ImageNet-Dogs Tiny-ImageNet WordNet
Benchmarks: STL10 Cifar10 Cifar100-20 ImageNet-Dogs Tiny-ImageNet
Metrics: ACC NMI ARI
Baselines: SIC MUST SCAN NNM PICA TCL CC GCC DCCM IIC DAC k-means
Builds on: CLIP SIC SCAN MUST
Closest papers by meaning
- Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency (AAAI 2026 · similarity 0.90)
- Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (AAAI 2024 · similarity 0.88)
- Hierarchical Semantic Alignment for Image Clustering (AAAI 2026 · similarity 0.88)
- Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers (NeurIPS 2026 · similarity 0.86)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.85)
- Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal Alignment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.