Research map / Vision-language and multimodal / CLIP and zero-shot

GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GET, which synthesizes pseudo text embeddings for unlabelled images from CLIP visual features (Text Embedding Synthesizer) and jointly trains visual and text branches with a cross-modal instance consistency loss for generalized category discovery.

Problem

Generalized category discovery on unlabelled data containing both known and novel classes, where visual-only representations struggle with visually similar classes and class names for unlabelled data are unavailable to use CLIP's text encoder.

Approach

A Text Embedding Synthesizer maps CLIP image embeddings through a linear layer into pseudo tokens for CLIP's text encoder, trained with a CLIP-style align loss and a distillation loss on labeled class-name text embeddings; a dual-branch (visual and text) parametric GCD framework then exchanges knowledge via a cross-modal instance consistency (KL) loss over labeled-class anchors.

Key results

• GET reaches 77.0% All accuracy on CUB, 78.5% on Stanford Cars and 58.9% on FGVC-Aircraft, surpassing SimGCD-CLIP by 5.3, 8.5 and 4.6 points on All classes.
• On ImageNet-R, GET improves All accuracy from 54.9% (SimGCD-CLIP) to 58.1%, and reports 3.2% and 6.0% gains over the prior SOTA for All and Old classes.
• Ablation on Stanford Cars shows TES adds 6.2 points on All accuracy and CICO adds about 11 points on New accuracy relative to the SimGCD-CLIP baseline.

Measured effects reported in the paper

Surprising findings

• Captioning-based pseudo text (BLIP, BLIP-2) performs far worse than the visual-only baseline on CUB (40.5% and 42.6% All), since captions describe pose and scene rather than class identity.
• Plain CLIP-based GCD baseline (GCD-CLIP) underperforms the DINO-based GCD on CIFAR100 novel classes, attributed to CLIP's weak zero-shot CIFAR100 performance.

Limitations

• Treats visual and text modalities as equally important, without adaptive per-sample modality weighting.
• Relies on CLIP, whose pretraining data may overlap with the unknown classes of GCD benchmarks.

Experimental setup

Datasets: CIFAR10 CIFAR100 ImageNet-100 CUB Stanford Cars FGVC-Aircraft Herbarium 19 ImageNet-R ImageNet-1K TV-100 NEV

Benchmarks: CIFAR10 CIFAR100 ImageNet-100 CUB Stanford Cars FGVC-Aircraft Herbarium 19 ImageNet-R ImageNet-1K TV-100 NEV

Metrics: clustering accuracy (ACC)

Baselines: GCD-CLIP SimGCD-CLIP SimGCD GCD PromptCAL µGCD LegoGCD CLIP-GCD TextGCD RS+ UNO+ ORCA

Builds on: SimGCD GCD BARON CLIP mutual knowledge distillation (Dual ranking statistics)

Closest papers by meaning

• Less Attention is More: Prompt Transformer for Generalized Category Discovery (CVPR 2025 · similarity 0.87)
• SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.87)
• CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.86)
• SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning (ICLR 2024 · similarity 0.86)
• SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery (NeurIPS 2025 · similarity 0.86)
• When Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach (CVPR 2025 · similarity 0.86)