Research map / Vision-language and multimodal / Human and embodied modeling
Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Multi-Sub performs customized multiple clustering by learning CLIP-based proxy words, expressed as weighted combinations of LLM-suggested reference words (GPT-4), jointly with an end-to-end clustering loss.
Problem
Users want clusterings of images tailored to a high-level interest (e.g., color or species), but existing deep multiple clustering methods require manual inspection of many clusterings, and prior CLIP-based approach Multi-MaP needs a contrastive concept and separates representation learning from clu
Approach
Represent each image's proxy word as a learned softmax-weighted combination of token embeddings of GPT-4-suggested reference words for the user's concept, train only the CLIP image projection layer with an alternating proxy-learning phase and an intra/inter-cluster loss, so representation learning and clustering are optimized end-to-end.
Key results
- Multi-Sub achieves NMI 0.9693 on Fruit color and 0.6654 on Fruit360 color, outperforming Multi-MaP (0.8619 and 0.6239) in the reported comparison.
- On CMUface identity, Multi-Sub reaches NMI 0.7441 and RI 0.9834 versus 0.6625 and 0.9496 for Multi-MaP.
- Multi-Sub surpasses CLIP_GPT in all cases and even CLIP_label in all reported settings, per the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Multi-Sub instead of the strongest listed baseline | Visual multiple clustering · Fruit color, Fruit360 color, Card order and suits, CMUface emotion/glass/identity/pose, Stanford Cars color/type, Flowers color/species, and CIFAR-10 type/envir | NMI: Multi-Sub is higher on 13 of 14 tasks; Fruit Species: 1.0000 vs 1.0000 | Helped |
| Use Multi-Sub instead of CLIP_GPT zero-shot clustering | Multi-Sub · All datasets and clustering tasks in the variants-of-CLIP table | NMI | Helped |
| Use Multi-Sub instead of CLIP_label zero-shot clustering | Multi-Sub · All datasets and clustering tasks in the variants-of-CLIP table | NMI | Helped |
| Replace CLIP text encoder with ALIGN (ablation) | Multi-Sub · Most tasks in the text-encoder comparison | NMI | Mixed |
| Replace CLIP text encoder with BLIP (ablation) | Multi-Sub · Text-encoder comparison | NMI | Mixed |
| Use proxy-word token embeddings as the subspace representation and cluster with their concatenation with image embeddings (ablation) | Multi-Sub · CIFAR-10 Type and Environment | NMI: 0.5271; 0.4828 | Helped |
| Use prompt embeddings t_i* rather than proxy-word token embeddings (ablation) | Multi-Sub · CIFAR-10 Type and Environment | NMI: 0.5013 vs 0.5271; 0.4514 vs 0.4828 | Hurt |
| Use text and image embeddings together rather than image embeddings alone (ablation) | Multi-Sub · CIFAR-10 Type and Environment | NMI: 0.5271 vs 0.3715; 0.4828 vs 0.4320 | Helped |
| Set the clustering-loss balance λ to 0.5 (ablation) | Multi-Sub · CIFAR-10 Type and Environment | NMI | Helped |
| Introduce a new shape preference and learn clustering in its subspace | Multi-Sub · Fruit shape | NMI: 0.717 -> 0.752 | Helped |
Surprising findings
- Multi-Sub outperforms CLIP with ground-truth labels (CLIP_label) in all reported settings, which the authors attribute to learned proxy embeddings.
- Using the word token embedding for clustering generally works better than the full prompt embedding.
Limitations
- Reliance on GPT-4 may introduce biases affecting clustering outcomes.
- Multiple clustering field lacks large, diverse datasets, limiting evaluation.
- Only one hyperparameter (lambda) but results depend on reference words; for identity-type tasks random WordNet words are used.
Experimental setup
Datasets: Stanford Cars Card CMUface Flowers Fruit Fruit360 CIFAR-10 WordNet
Benchmarks: Stanford Cars Card CMUface Flowers Fruit Fruit360 CIFAR-10
Metrics: NMI RI
Baselines: MSC MCV ENRC iMClusts AugDMC DDMC Multi-MaP CLIP_GPT CLIP_label
Builds on: CLIP Multi-MaP AugDMC
Closest papers by meaning
- Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering (CVPR 2024 · similarity 0.91)
- Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering (AAAI 2026 · similarity 0.90)
- Multi-View Multiple Clusterings Using Deep Matrix Factorization (AAAI 2020 · similarity 0.88)
- Exploring a Principled Framework for Deep Subspace Clustering (ICLR 2025 · similarity 0.86)
- Contrastive Multi-view Subspace Clustering via Tensor Transformers Autoencoder (AAAI 2025 · similarity 0.86)
- Medusa: A Multi-Scale High-order Contrastive Dual-Diffusion Approach for Multi-View Clustering (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.