Research map / Vision-language and multimodal / Human and embodied modeling
Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-MaP, which learns a CLIP proxy word embedding guided by GPT-4 reference words and concept constraints to produce user-interest-specific clusterings of unlabeled images.
Problem
Deep multiple clustering yields many clusterings, but users need only the one matching their interest; mapping a short user keyword to the right visual aspect of unlabeled images is hard without labels.
Approach
Learns a continuous proxy word embedding inserted into a CLIP text prompt ("a fruit with the color of *") by maximizing image-text similarity, constrained toward a user concept word and a GPT-4-suggested nearest reference word, with an optional contrastive term against irrelevant concepts.
Key results
- Multi-MaP outperforms all five deep and traditional multiple clustering baselines on every listed clustering task in Table 2, e.g., Fruit360 color NMI 0.6239 vs 0.4594 for AugDMC.
- On Identity clustering of CMUface, Multi-MaP reaches NMI 0.6625 and RI 0.9496, versus 0.5875 and 0.8334 for AugDMC.
- Ablation shows concept-word and reference-word constraints each improve over proxy learning alone, and adding contrastive concepts gives the best results in all cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add concept-level constraint to proxy learning (ablation) | Multi-MaP · Fruit color | NMI: 0.7642 -> 0.8215 | Helped |
| add reference word constraint to proxy learning (ablation) | Multi-MaP · Fruit color | NMI: 0.7642 -> 0.8136 | Helped |
| combine concept word and reference word constraints (ablation) | Multi-MaP · Fruit color | NMI: 0.8215 -> 0.8484 | Helped |
| add contrastive concepts constraint (ablation) | Multi-MaPcr · Fruit color | NMI: 0.8484 -> 0.8619 | Helped |
| increase reference word constraint weight α to α=0.4 (ablation) | Multi-MaP · Fruit color | NMI | Helped |
| increase concept-level constraint weight β to β=0.3 (ablation) | Multi-MaP · Fruit color | NMI | Helped |
| use Multi-MaP instead of the strongest listed baseline | visual multiple clustering methods · Fruit color | NMI: 0.8517 -> 0.8619 | Helped |
| use Multi-MaP instead of the strongest listed baseline | visual multiple clustering methods · Stanford Cars color | NMI: 0.2736 -> 0.7360 | Helped |
| use Multi-MaP instead of CLIPGPT zero-shot clustering | CLIP · CMUface identity | NMI: 0.1998 -> 0.6625 | Helped |
| use Multi-MaP instead of CLIPlabel zero-shot clustering | CLIP · CMUface identity | NMI: 0.2677 -> 0.6625 | Helped |
| use Multi-MaP instead of CLIPlabel zero-shot clustering | CLIP · CMUface glass | NMI: 0.4686 -> 0.3402 | Hurt |
Surprising findings
- Multi-MaP beats CLIP zero-shot with ground-truth labels (CLIPlabel) on some tasks, such as color clustering on Fruit360.
Limitations
- Relies on data having semantically meaningful labels that GPT-4 or WordNet can supply.
- Requires the user to supply a concept keyword; performance depends on candidate reference word quality, which may be noisy.
- Evaluated only on image datasets with ground-truth clusterings, with NMI and RI as the only metrics.
Experimental setup
Datasets: ALOI Card CMUface Fruit Fruit360 Stanford Cars Flowers
Benchmarks: ALOI Fruit Fruit360 Card CMUface Stanford Cars Flowers
Metrics: NMI RI
Baselines: MSC MCV ENRC iMClusts AugDMC CLIP zero-shot (GPT-4 candidate labels) CLIP zero-shot (ground-truth labels)
Builds on: CLIP CoOp COALA
Closest papers by meaning
- Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning (NeurIPS 2024 · similarity 0.91)
- Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers (NeurIPS 2026 · similarity 0.86)
- Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models (ICLR 2024 · similarity 0.86)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.85)
- Multi-Level Cross-Modal Alignment for Image Clustering (AAAI 2024 · similarity 0.85)
- MMRL: Multi-Modal Representation Learning for Vision-Language Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.