Research map / Vision-language and multimodal / Human and embodied modeling

Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Language-grounded scene text and structure · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a language-assisted image clustering framework that builds a cross-modal image-text representation matrix via ridge regression over WordNet-derived text features (K-means on it as a baseline), then learns continuous per-class semantic centers via prompt learning on frozen CLIP with neighbor-consistency pseudo-label filtering.

Problem

Existing language-assisted image clustering (LAIC) methods build text features for images that are highly similar across samples (weak inter-class discriminability) and restrict clustering to pre-built image-text alignments, underusing the VLM's native semantic space.

Approach

Reconstruct image features from the whole WordNet-derived candidate text set via closed-form ridge regression to obtain a cross-modal relational representation matrix, then learn fixed-prefix continuous semantic centers by prompt learning on high-confidence pseudo-labels, assigning test images to the nearest center.

Key results

• Averages 2.6% improvement over state-of-the-art LAIC method TAC across eight benchmarks, with ImageNet-Dogs ACC 88.0 vs TAC 83.0.
• On CIFAR-10 the method reaches 92.9 ACC, 85.2 NMI, and an average of 92.5 across the four standard datasets (per the table).
• Ablation on ImageNet-Dogs and DTD shows the full loss combination (sup+con+ent) gives the best results, e.g., DTD ACC 53.7.

Surprising findings

• Learned semantic centers can outperform zero-shot CLIP using ground-truth class names, with the paper reporting an average 5.8% gain over ZS-CLIP.
• The consistency and entropy losses alone yield very poor clustering (e.g., ACC around 10-17% on ImageNet-Dogs), only helping when combined with the supervised loss.

Limitations

• Relies on WordNet noun candidate set and hyperparameters (theta, gamma, k-hat) tuned differently for datasets with many clusters.
• Evaluated mainly on CLIP ViT-B/32; generalization to other VLMs shown only briefly with BLIP on three datasets.
• Clustering-of-results comparison reports average gains mostly over prior LAIC methods, with reliance on CLIP's text-image alignment.

Experimental setup

Datasets: ImageNet-Dogs ImageNet-10 STL-10 CIFAR-10 Flowers-102 CIFAR-100 DTD UCF-101 WordNet

Benchmarks: ImageNet-Dogs ImageNet-10 STL-10 CIFAR-10 Flowers-102 CIFAR-100 DTD UCF-101

Metrics: NMI ACC ARI

Baselines: TAC GradNorm SIC MCA PRO-DSC LFSS ZS-CLIP SCAN TCL BYOL

Builds on: CLIP TAC SIC WordNet generalized cross-entropy

Closest papers by meaning

• Image Clustering with External Guidance (ICML 2024 · similarity 0.90)
• Image Clustering Conditioned on Text Criteria (ICLR 2024 · similarity 0.89)
• Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models (ICLR 2024 · similarity 0.88)
• Target Semantics Clustering via Text Representations for Robust Universal Domain Adaptation (AAAI 2025 · similarity 0.88)
• Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network (AAAI 2024 · similarity 0.87)
• Remodeling Semantic Relationships in Vision-Language Fine-Tuning (AAAI 2026 · similarity 0.87)