Research map / Label-efficient and robust learning / Recognition and long-tail
Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GNM-PT, a visual prompt tuning method for long-tailed recognition that replaces SAM's gradient-based worst-case perturbation with a random Gaussian parameter perturbation, giving a flatter loss landscape at lower computational cost.
Problem
Visual prompt tuning (VPT) generalizes poorly to tail classes in long-tailed visual recognition; SAM-style flat-minima methods are head-class dominated and require two gradient computations per step.
Approach
Replace SAM's max-loss neighborhood with a Gaussian neighborhood loss (expected loss under random Gaussian parameter perturbations), sampled independently of inputs so it avoids head-class-dominated perturbations and the extra gradient pass; combined with DRW re-balancing and a merge-prompt classification head.
Key results
- GNM-PT reaches 90.3% top-1 on CIFAR100-LT with IR 100, 76.5% on iNaturalist 2018, and 50.1% on Places-LT, reported as state of the art.
- On CIFAR100-LT, GNM-PT improves over LPT by 1.3, 1.2, 1.2 and 0.8 points at imbalance ratios 200, 100, 50 and 10.
- GNM adds under 2 seconds per epoch, while SAM takes more than 1.8x the baseline training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DRW to VPT training (ablation) | VPT on CIFAR100-LT (IR 100) · CIFAR100-LT (IR 100) | Accuracy: 88.25 -> 89.32 | Helped |
| add merge prompt to DRW-trained VPT (ablation) | VPT on CIFAR100-LT (IR 100) · CIFAR100-LT (IR 100) | Accuracy: 89.32 -> 89.58 | Helped |
| add GNM optimization (ablation) | VPT with DRW and merge prompt · CIFAR100-LT (IR 100) | Accuracy: 89.58 -> 90.28 | Helped |
| use GNM-PT instead of LPT | ViT-B/16 · CIFAR100-LT (IR 100) | Top-1 accuracy: 89.1 -> 90.3 | Helped |
| use GNM-PT instead of LPT | ViT-B/16 · Places-LT | Overall accuracy: 49.7 -> 50.1 | Helped |
| use GNM-PT instead of GML | ViT-B/16 · ImageNet-LT | Overall accuracy: 78.0 -> 80.4 | Helped |
| add DRW to GNM-PT (ablation) | GNM-PT · iNaturalist 2018 | Overall accuracy: 76.3 -> 76.5 | Helped |
| add DRW to GNM-PT (ablation) | GNM-PT · Places-LT | Overall accuracy: 50.0 -> 50.1 | No clear effect |
| add DRW to GNM-PT (ablation) | GNM-PT · Places-LT | Head accuracy: 48.6 -> 46.6 | Hurt |
| increase GNM amplitude to a=2.0 (ablation) | GNM-PT · CIFAR100-LT (IR 100) | Accuracy: 90.28 -> 88.45 | Hurt |
| replace Gaussian perturbations with uniform perturbations (ablation) | GNM-PT · CIFAR100-LT (IR 100) | Accuracy: (90.28) -> (90.17) | No clear effect |
| use GNM instead of SAM | VPT with GCL on CIFAR100-LT (IR 100) · CIFAR100-LT (IR 100) | Tail accuracy | Helped |
Surprising findings
- Random Gaussian perturbation outperformed the theoretically optimal SAM perturbation (90.28% vs 89.69% on CIFAR100-LT IR 100).
- SAM reduced tail-class accuracy for GCL in the reported analysis, while GNM improved all classes.
Limitations
- Head-class accuracy drops when DRW re-balancing is applied, since the classifier re-balancing trades head performance for tail performance.
- Gains over SAM-based and LPT methods are modest on iNat and Places-LT, and GNM does not beat ImbSAM in stage 1 on tail classes.
- The choice of effective-number parameters in DRW may be suboptimal, and the perturbation distribution and amplitude are set by heuristic ablation.
Experimental setup
Datasets: CIFAR100-LT iNaturalist 2018 Places-LT ImageNet-LT
Benchmarks: CIFAR100-LT iNaturalist 2018 Places-LT ImageNet-LT
Metrics: top-1 accuracy head/medium/tail accuracy training time per epoch
Baselines: VPT LPT LiVT SAM CCSAM ImbSAM VL-LTR RAC GPaCo SHIKE NCL RIDE
Builds on: visual prompt tuning sharpness-aware minimization LPT GCL deferred re-weighting
Closest papers by meaning
- LPT: Long-tailed Prompt Tuning for Image Classification (ICLR 2023 · similarity 0.90)
- Global and Local Mixture Consistency Cumulative Learning for Long-Tailed Visual Recognitions (CVPR 2023 · similarity 0.88)
- Decoupled Optimisation for Long-Tailed Visual Recognition (AAAI 2024 · similarity 0.88)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.87)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.87)
- SA²VP: Spatially Aligned-and-Adapted Visual Prompt (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.