Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPTNet, a two-stage alternating framework that tunes spatial pixel-level prompts (data parameters) and the last ViT block plus head (model parameters) for generalized category discovery.
Problem
Generalized category discovery requires classifying unlabelled images from seen and unseen classes; full fine-tuning of pre-trained models overfits and is costly, while joint optimization of model and data (prompt) parameters is unstable and sub-optimal.
Approach
Alternate between learning instance-agnostic spatial prompts wrapped around each image patch in pixel space (with the backbone frozen) and fine-tuning the last transformer block and projection head (with prompts frozen), using the prompted image as a learned augmentation within contrastive and self-distillation losses, built on SimGCD.
Key results
- SPTNet reaches 61.4% average All accuracy on SSB, versus 56.1% for SimGCD, with 0.117% extra parameters relative to ViT-B.
- On generic datasets SPTNet achieves 97.3 (CIFAR-10), 81.3 (CIFAR-100) and 85.4 (ImageNet-100) All accuracy, improving over SimGCD by 0.4, 1.9 and 2.5 points.
- Alternating prompt and model optimization (61.4 SSB All) outperforms end-to-end (58.6), data-first (58.0) and model-first (59.2) training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VPT prompts to SimGCD (ablation) | SimGCD · SSB All | accuracy: 56.1 -> 54.4 | Hurt |
| add a global prompt to SimGCD (ablation) | SimGCD · SSB All | accuracy: 56.1 -> 56.7 (+0.6) | Helped |
| add spatial prompts to SimGCD (ablation) | SimGCD · SSB All | accuracy: 56.1 -> 57.9 (+1.8) | Helped |
| alternate prompt and model optimization and add a global prompt (ablation) | SimGCD + spatial prompts · SSB All | accuracy: 56.1 -> 61.4 (+5.3) | Helped |
| use one shared spatial prompt across patches instead of distinct patch prompts (ablation) | SPTNet · SSB All | accuracy: 56.1 -> 60.9 (+4.8) | Helped |
| remove the global prompt, retaining distinct spatial prompts and alternative training (ablation) | SPTNet · SSB All | accuracy: 61.4 -> 59.1 | Hurt |
| jointly train prompts and model end-to-end instead of alternating (ablation) | SPTNet · SSB All | accuracy: 61.4 -> 58.6 | Hurt |
| train prompts first, then model, instead of alternating (ablation) | SPTNet · SSB All | accuracy: 61.4 -> 58.0 | Hurt |
| train model first, then prompts, instead of alternating (ablation) | SPTNet · SSB All | accuracy: 61.4 -> 59.2 | Hurt |
| increase spatial prompt size (ablation) | SPTNet · SSB | accuracy | Hurt |
| vary the alternation interval k (ablation) | SPTNet · SSB | accuracy | No clear effect |
| add SPTNet to the ViT-B/16 baseline | GCD on CIFAR-10, CIFAR-100, and ImageNet-100 · CIFAR-10 All; CIFAR-100 All; ImageNet-100 All | accuracy: +0.4%; +1.9%; +2.5% | Helped |
Surprising findings
- Naively applying VPT on top of SimGCD reduces accuracy (56.1 to 54.4 on SSB), showing hidden-space prompts hurt GCD.
- End-to-end joint optimization of prompts and model leads to most prompts deactivating, suggesting a shortcut solution.
Limitations
- Gains on low-resolution CIFAR-10/100 are limited because each patch carries little information.
- Gains on ImageNet-100 are small because the DINO backbone is already highly tuned for that dataset.
- Hyperparameters (learning rates, prompt sizes) vary per dataset and were tuned on validation splits.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 CUB Stanford Cars FGVC-Aircraft Herbarium 19 DomainNet
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 CUB Stanford Cars FGVC-Aircraft Herbarium 19 DomainNet
Metrics: clustering accuracy (ACC) All/Old/New
Baselines: k-means RankStats+ UNO+ GCD ORCA SimGCD DCCL PromptCAL VPT
Builds on: SimGCD VPT DINO EM algorithm
Closest papers by meaning
- Less Attention is More: Prompt Transformer for Generalized Category Discovery (CVPR 2025 · similarity 0.92)
- Solving the Catastrophic Forgetting Problem in Generalized Category Discovery (CVPR 2024 · similarity 0.89)
- Transfer and Alignment Network for Generalized Category Discovery (AAAI 2024 · similarity 0.88)
- HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts (ICLR 2025 · similarity 0.88)
- DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discovery (ICLR 2025 · similarity 0.87)
- Hyperbolic Category Discovery (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.