Research map / Vision-language and multimodal / Human and embodied modeling
Less Attention is More: Prompt Transformer for Generalized Category Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AptGCD, which adds a Meta Visual Prompt generator (recurrently self-generated from ViT intermediate features) and a Prompt Transformer (prompts as learnable queries in attention) to a DINO ViT-B/16 for generalized category discovery, tuning only the last block and a small prompt generator.
Problem
Generalized category discovery on fine-grained and general images suffers because ViT global attention lacks fine-grained inner-patch local modeling and local-global interaction, so attention scatters to background and novel classes are poorly recognized.
Approach
A task-independent meta prompt generator with a recurrent dual-forward self-generation mechanism produces local prompts that are added to intermediate ViT features, and a Prompt Transformer injects these prompts as learnable queries alongside global QKV attention so local details are balanced with global context.
Key results
- AptGCD reaches 70.3% All and 69.2% New on CUB-200, versus 63.8% All for LegoGCD, and averages about 9.2% proportional / 4.0% absolute New-accuracy gain over SPTNet on the four fine-grained datasets.
- On ImageNet-100 AptGCD achieves 87.8% All accuracy versus 86.3% for LegoGCD; on CIFAR-100 it reaches 82.8% All and 85.5% New.
- Plugging MVP and PT into LegoGCD raises SSB-average New accuracy from 53.3% to 60.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AptGCD instead of SPTNet | GCD · CUB-200 New | accuracy: 65.1 -> 69.2 | Helped |
| use AptGCD instead of SPTNet | GCD · Stanford Cars New | accuracy: 49.3 -> 53.6 | Helped |
| use AptGCD instead of SPTNet | GCD · FGVC-Aircraft New | accuracy: 58.1 -> 59.0 | Helped |
| use AptGCD instead of LegoGCD | GCD · Herbarium19 New | accuracy: 38.4 -> 42.3 | Helped |
| use AptGCD instead of LegoGCD | GCD · ImageNet-100 New | accuracy: 82.1 -> 84.3 | Helped |
| use AptGCD instead of LegoGCD | GCD · CIFAR-100 New | accuracy: 82.5 -> 85.5 | Helped |
| use AptGCD instead of SPTNet | GCD · CIFAR-10 New | accuracy: 98.6 -> 98.7 | Helped |
| replace random VPT prompting with MVP (ablation) | AptGCD · SSB-Avg New | accuracy: 51.5 -> 57.3 | Helped |
| replace Global prompting with MVP (ablation) | AptGCD · SSB-Avg New | accuracy: 54.5 -> 57.3 | Helped |
| replace SPM prompting with MVP (ablation) | AptGCD · SSB-Avg New | accuracy: 55.7 -> 57.3 | Helped |
| use PT interactions in the last three blocks instead of Block 12 only (ablation) | AptGCD · SSB-Avg New | accuracy: 58.2 -> 60.6 | Helped |
| jointly train PG and PT instead of training them separately (ablation) | AptGCD · SSB-Avg New | accuracy: 57.3 -> 60.6 | Helped |
Surprising findings
- Standard VPT visual prompts reduce GCD performance on SSB (All 58.7 to 56.5), while the proposed MVP improves it to 62.3.
Limitations
- Gains on fine-grained data are large, but the method's advantage on general datasets (CIFAR-10, CIFAR-100 All accuracy) is smaller or mixed, with lower Old accuracy than GPC on CIFAR-100.
- Prompt transformer interaction is applied to only the last three blocks, chosen for compute constraints, so the full design space was not explored.
Experimental setup
Datasets: CUB-200 Stanford Cars FGVC-Aircraft Herbarium 19 CIFAR-10 CIFAR-100 ImageNet-100
Benchmarks: CUB-200 Stanford Cars FGVC-Aircraft Herbarium 19 CIFAR-10 CIFAR-100 ImageNet-100
Metrics: clustering accuracy (ACC) New accuracy Old accuracy All accuracy
Baselines: GCD GPC PIM SimGCD µGCD DCCL PromptCAL SPTNet CiPR CMS LegoGCD VPT
Builds on: SimGCD LegoGCD VPT DINO SPTNet
Closest papers by meaning
- SPTNet: An Efficient Alternative Framework for Generalized Category Discovery with Spatial Prompt Tuning (ICLR 2024 · similarity 0.92)
- Solving the Catastrophic Forgetting Problem in Generalized Category Discovery (CVPR 2024 · similarity 0.87)
- GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery (CVPR 2025 · similarity 0.87)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.87)
- When Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach (CVPR 2025 · similarity 0.86)
- HIDISC: A Hyperbolic Framework for Domain Generalization with Generalized Category Discovery (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.