Research map / Vision-language and multimodal / Cross-modal learning
Image Clustering with External Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes externally guided clustering, using WordNet nouns retrieved via CLIP as text counterparts for images plus cross-modal mutual distillation of neighborhood assignments (TAC), reaching SOTA on eight image clustering benchmarks.
Problem
Image clustering is limited by internal supervision signals derived only from the image data; external knowledge such as semantic descriptions is unused when class names are unavailable.
Approach
Treat external knowledge (WordNet nouns) as a new supervision signal: select discriminative nouns for k-means semantic centers, retrieve a soft text counterpart per image with CLIP, concatenate with image features, and train two cluster heads with a cross-modal mutual distillation loss over nearest neighbors.
Key results
- TAC achieves 83.2% average over five classic datasets (e.g., ImageNet-Dogs ACC 83.0, ARI 72.2), surpassing zero-shot CLIP (78.7 avg).
- On DTD, UCF-101 and ImageNet-1K, TAC reaches 59.9 average vs. 56.7 for zero-shot CLIP and 54.6 for SIC; ImageNet-1K ACC 58.2 vs. 63.6 for zero-shot CLIP.
- Training-free TAC (no train) improves CLIP k-means ARI by 14.4% on CIFAR-20 and 43.5% on ImageNet-Dogs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add WordNet text counterparts and cross-modal mutual distillation to CLIP k-means | CLIP image clustering · ImageNet-Dogs val | ACC: 38.1 -> 83.0 | Helped |
| Add WordNet text counterparts and cross-modal mutual distillation to CLIP k-means | CLIP image clustering · UCF-101 val | ACC: 58.2 -> 68.7 | Helped |
| Add WordNet text counterparts and cross-modal mutual distillation to CLIP k-means | CLIP image clustering · ImageNet-1K val | ACC: 38.9 -> 58.2 | Helped |
| Add WordNet text counterparts and cross-modal mutual distillation | CLIP image clustering · ImageNet-1K val | ACC: 63.6 -> 58.2 | Hurt |
| Add cross-modal mutual distillation training to text-counterpart k-means | TAC (no train) · ImageNet-Dogs val | ACC: 75.1 -> 83.0 | Helped |
| Add cross-modal mutual distillation training to text-counterpart k-means | TAC (no train) · DTD test | ACC: 45.9 -> 50.1 | Helped |
| Remove L_Bal (ablation) | TAC · ImageNet-Dogs val | ACC: 83.0 -> 57.0 | Hurt |
| Remove L_Bal (ablation) | TAC · UCF-101 val | ACC: 68.7 -> 3.6 | Hurt |
| Remove L_Con (ablation) | TAC · UCF-101 val | ACC: 68.7 -> 45.1 | Hurt |
| Remove L_Dis (ablation) | TAC · ImageNet-Dogs val | ACC: 83.0 -> 81.8 | Hurt |
| Remove L_Dis (ablation) | TAC · UCF-101 val | ACC: 68.7 -> 67.3 | Hurt |
| Use text-to-image rather than bidirectional distillation (ablation) | TAC · ImageNet-Dogs val | ACC: 83.0 -> 82.1 | Hurt |
Surprising findings
- Manually annotated class names are not always the best semantic description; retrieved WordNet nouns can separate visually similar classes better than class names (e.g., zero-shot CLIP confuses Blenheim Spaniel).
- Retrieval from all WordNet nouns without discriminative noun selection degrades performance, showing the selection step matters.
Limitations
- Requires manually setting the target cluster number K and a granularity parameter (k = N/300).
- Zero-shot CLIP with class names remains better on ImageNet-1K (ACC 63.6 vs 58.2 for TAC).
- Evaluated only on image data with a single external source (WordNet) and a single VLP model (CLIP).
Experimental setup
Datasets: STL-10 CIFAR-10 CIFAR-20 ImageNet-10 ImageNet-Dogs DTD UCF-101 ImageNet-1K WordNet
Benchmarks: STL-10 CIFAR-10 CIFAR-20 ImageNet-10 ImageNet-Dogs DTD UCF-101 ImageNet-1K
Metrics: NMI ACC ARI
Baselines: CLIP (k-means) CLIP (zero-shot) SIC SCAN SPICE TCC GCC CC IDFD DAC DCCM PICA
Builds on: CLIP SCAN k-means WordNet
Closest papers by meaning
- Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers (NeurIPS 2026 · similarity 0.90)
- Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models (ICLR 2024 · similarity 0.87)
- Image Clustering Conditioned on Text Criteria (ICLR 2024 · similarity 0.87)
- RPSC: Robust Pseudo-Labeling for Semantic Clustering (AAAI 2024 · similarity 0.86)
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.85)
- Semantic-Guided Novel Category Discovery (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.