Research map / Label-efficient and robust learning / Recognition and long-tail
Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPP, a CLIP-based image clustering pipeline that refines frozen CLIP features with the MCR² rate-reduction objective (MLC), estimates the number of clusters via a coding-length criterion without retraining, and captions clusters using CLIP image-text matching.
Problem
Clustering unlabeled images at large scale (e.g., ImageNet-1k, uncurated datasets) remains poor, and the number of clusters is often unknown.
Approach
Integrate frozen CLIP ViT-L/14 features into the MLC rate-reduction framework (with a diversifying warm-up on the rate term and a doubly stochastic membership via Sinkhorn), estimate the cluster count from coding length of features plus labels without retraining, and caption clusters by CLIP zero-shot voting over an LLM-generated and ImageNet label vocabulary.
Key results
- On ImageNet-1k, CPP reaches 66.2% ACC and 86.8% NMI versus 64.0% for TEMI and 49.2% for KMeans on CLIP features.
- CPP achieves 97.4% ACC on CIFAR-10, 64.2% on CIFAR-20, and 74.0% on CIFAR-100, outperforming TEMI and other listed deep clustering methods.
- The coding-length criterion recovers 10 clusters on CIFAR-10 and 20 on CIFAR-100, and selects 150 for MS-COCO and 300 for LAION-Aesthetics without retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TEMI with CPP | CLIP ViT-L/14 clustering · CIFAR-10 | ACC: 96.9 -> 97.4 | Helped |
| replace TEMI with CPP | CLIP ViT-L/14 clustering · CIFAR-20 | NMI: 64.5 -> 72.5 | Helped |
| replace TEMI with CPP | CLIP ViT-L/14 clustering · CIFAR-100 | ACC: 73.7 -> 74.0 | Helped |
| replace TEMI with CPP | CLIP ViT-L/14 clustering · ImageNet-1k | ACC: 64.0 -> 66.2 | Helped |
| add diversified initialization by optimizing the feature-volume objective (ablation) | CPP · CIFAR-10 | ACC: 87.6 -> 97.4 | Helped |
| add diversified initialization by optimizing the feature-volume objective (ablation) | CPP · CIFAR-20 | ACC: 57.4 -> 64.2 | Helped |
| add diversified initialization by optimizing the feature-volume objective (ablation) | CPP · CIFAR-100 | ACC: 67.9 -> 74.0 | Helped |
| add diversified initialization by optimizing the feature-volume objective (ablation) | CPP · ImageNet-1k | ACC: 63.7 -> 66.2 | Helped |
| use OpenCLIP pretrained on LAION-2B instead of OpenAI CLIP (ablation) | CPP · CIFAR-10 | ACC: 97.4 -> 96.5 | Hurt |
| use OpenCLIP pretrained on LAION-2B instead of OpenAI CLIP (ablation) | CPP · CIFAR-20 | ACC: 64.2 -> 70.2 | Helped |
| use OpenCLIP pretrained on LAION-2B instead of OpenAI CLIP (ablation) | CPP · CIFAR-100 | ACC: 74.0 -> 76.7 | Helped |
| use OpenCLIP pretrained on LAION-2B instead of OpenAI CLIP (ablation) | CPP · ImageNet-1k | ACC: 66.2 -> 66.8 | Helped |
Surprising findings
- MAE features perform poorly for clustering (24.2% ACC on ImageNet-1k with CPP), while DINO and CLIP give good initializations.
- On ImageNet-1k, KMeans on DINO ViT-B/8 (56.0%) slightly beats CPP on DINO ViT-B/8 in ACC? Actually CPP gives 61.9% there, but the KMeans baseline with DINO ViT-B/8 is higher in ACC than the CPP ViT-B/16 variant.
Limitations
- Relies on a frozen CLIP backbone, so clustering quality is bounded by the pre-trained features.
- Captioning uses a fixed candidate vocabulary (LLM-generated labels plus ImageNet labels) rather than open-ended text.
- Number of clusters selection and hyperparameters (gamma, d) are set per dataset.
Experimental setup
Datasets: CIFAR-10 CIFAR-20 CIFAR-100 ImageNet-1k MS-COCO LAION-Aesthetics WikiArt
Benchmarks: CIFAR-10 CIFAR-20 CIFAR-100 ImageNet-1k MS-COCO LAION-Aesthetics WikiArt
Metrics: clustering accuracy (ACC) normalized mutual information (NMI)
Baselines: MLC SCAN IMC-SWAV TEMI KMeans EnSC SSC-OMP Spectral Clustering SPICE RUC+SCAN
Builds on: Manifold Linearizing and Clustering (MLC) CLIP Maximal Coding Rate Reduction (MCR^2) Sinkhorn distance projection spectral clustering
Closest papers by meaning
- Semantic-Enhanced Image Clustering (AAAI 2023 · similarity 0.89)
- Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers (NeurIPS 2026 · similarity 0.88)
- Image Clustering Conditioned on Text Criteria (ICLR 2024 · similarity 0.88)
- Image Clustering with External Guidance (ICML 2024 · similarity 0.87)
- RPSC: Robust Pseudo-Labeling for Semantic Clustering (AAAI 2024 · similarity 0.86)
- Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.