Research map / Vision-language and multimodal / Human and embodied modeling
Image Clustering Conditioned on Text Criteria
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IC|TC, a training-free pipeline that clusters images under a user-specified text criterion by chaining a VLM (LLaVA) for image descriptions with an LLM (GPT-4) for cluster naming and assignment.
Problem
Classical image clustering gives users no direct control over the clustering criterion, and results often disagree with the grouping a user has in mind (e.g., action, location, mood instead of foreground object).
Approach
Use a vision-language model to produce criterion-conditioned text descriptions of each image, then an instruction-tuned LLM to discover K cluster names from the raw labels and assign each description to a cluster, with the user iteratively refining the text criterion.
Key results
- On CIFAR-10 with criterion Object, IC|TC (BLIP-2 + GPT-4) reaches 0.975 ACC vs 0.926 for SPICE, the best classical baseline.
- On Stanford 40 Action, IC|TC (LLaVA + GPT-4) reaches 0.774 ACC under the Action criterion, versus 0.397 for SCAN.
- On CIFAR-10-Gen (synthetic images), IC|TC reaches 98.7% accuracy, used to argue against data contamination.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use IC|TC with BLIP-2 + GPT-4 instead of SPICE | IC|TC · CIFAR-10 (Object) | ACC: 0.926 -> 0.975 | Helped |
| use IC|TC with BLIP-2 + GPT-4 instead of SPICE | IC|TC · STL-10 (Object) | ACC: 0.938 -> 0.993 | Helped |
| use IC|TC with LLaVA + GPT-4 instead of SPICE | IC|TC · CIFAR-100 (Object) | ACC: 0.584 -> 0.589 | No clear effect |
| use LLaVA + GPT-4 rather than LLaVA alone (ablation) | IC|TC · CIFAR-10 (Object) | ACC: 0.647 -> 0.910 | Helped |
| use LLaVA + GPT-4 rather than LLaVA alone (ablation) | IC|TC · STL-10 (Object) | ACC: 0.774 -> 0.986 | Helped |
| use LLaVA + GPT-4 rather than LLaVA alone (ablation) | IC|TC · CIFAR-100 (Object) | ACC: 0.097 -> 0.589 | Helped |
| use the text criterion in Steps 2 and 3 (ablation) | IC|TC · Stanford 40 Action (Action) | ACC: 0.152 -> 0.774 | Helped |
| replace LLaVA with ClipCap as the VLM (ablation) | IC|TC · Stanford 40 Action (Action) | ACC: 0.774 -> 0.250 | Hurt |
| replace LLaVA with BLIP-2 as the VLM (ablation) | IC|TC · Stanford 40 Action (Action) | ACC: 0.774 -> 0.427 | Hurt |
| use GPT-4 rather than GPT-3.5 (ablation) | IC|TC with LLaVA · CIFAR-100 (Object) | ACC: 0.513 -> 0.589 | Helped |
| use K=7 rather than K=2 (ablation) | IC|TC · PPMI (Musical Instrument) | accuracy: 96.4%; 97.7% | Mixed |
| add the prompt instructing the method not to consider gender | IC|TC · FACET (craftsman and laborer clusters) | gender ratio disparity: 27.2%→4.4%; 11.6%→3.2% | Helped |
Surprising findings
- LLaVA alone with a prompt performs much worse on CIFAR-100 (ACC 0.097) than the full pipeline, showing the LLM stage is essential.
- Removing the text criterion from Steps 2 and 3 drops Stanford 40 Action ACC from 0.774 to 0.152.
Limitations
- Depends on proprietary GPT-4 for best results; open-source Llama-2 variants perform slightly worse.
- Fairness and bias in foundation models can propagate into clusters; gender bias observed on FACET subset.
- Evaluation on Location and Mood relies on human labels for only 1000 sampled images, and the comparison with classical methods is arguably unfair.
Experimental setup
Datasets: Stanford 40 Action PPMI CIFAR-10 STL-10 CIFAR-100 FACET
Benchmarks: CIFAR-10 STL-10 CIFAR-100 Stanford 40 Action PPMI FACET CIFAR-10-Gen
Metrics: ACC NMI ARI
Baselines: SCAN SPICE RUC TCL IIC LLaVA only ClipCap BLIP-2 K-means on LLaVA embeddings
Builds on: LLaVA Stable Diffusion XL
Closest papers by meaning
- Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers (NeurIPS 2026 · similarity 0.89)
- Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models (ICLR 2024 · similarity 0.88)
- Hierarchical Semantic Alignment for Image Clustering (AAAI 2026 · similarity 0.88)
- Image Clustering with External Guidance (ICML 2024 · similarity 0.87)
- Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering (CVPR 2024 · similarity 0.84)
- RPSC: Robust Pseudo-Labeling for Semantic Clustering (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.