Research map / Vision-language and multimodal / CLIP and zero-shot
SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SharpZO, a two-stage forward-only VLM prompt-tuning method that warms up with a sharpness-aware CMA-ES and then runs sparse zeroth-order SGD with Z-pruning, reaching 75.64% average accuracy over 11 datasets on CLIP ViT-B/16.
Problem
Prompt tuning of vision-language models normally requires backpropagation, which is unavailable on memory-constrained inference-only edge devices; existing backprop-free methods (ES, ZO-SGD) have high variance and underperform.
Approach
A sharpness-aware CMA-ES warm-up (adding a SAM-style worst-case perturbation to CMA-ES sampling, with coordinate-wise gradient estimates) provides a strong, smoothed initialization, followed by sparse ZO-SGD with q=1 random perturbations masked by a Z-score-normalized Fisher-style pruning criterion.
Key results
- On 16-shot CLIP ViT-B/16 across 11 datasets, SharpZO reaches 75.64% average accuracy vs 70.98% for ZIP (reproduced) and 68.20% for BlackVIP, outperforming ZIP on all 11 tasks.
- On ImageNet distribution-shift benchmarks, SharpZO gains 2.0 points over ZIP on ResNet-50 and 2.8 points on ViT-B/16 (averaged).
- Time-to-test-accuracy on ImageNet is 15.3 min for SharpZO vs 19.0 min for ZIP and 172.6 min for BlackVIP; SharpZO uses 2 queries per step in stage 2 vs 10 for ZIP/BlackVIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ZIP with SharpZO | CLIP ViT-B/16 · 11-dataset few-shot average | accuracy: 70.98 -> 75.64 | Helped |
| replace ZIP with SharpZO | CLIP RN50 · 11-dataset few-shot average | accuracy: 64.08 -> 69.76 | Helped |
| replace ZIP with SharpZO | CLIP RN50 · ImageNet distribution-shift average | accuracy: 42.2 -> 43.3 | Helped |
| replace ZIP with SharpZO | CLIP ViT-B/16 · ImageNet distribution-shift average | accuracy: 56.9 -> 58.9 | Helped |
| replace CraFT with SharpZO | CLIP ViT-B/16 · EuroSAT | accuracy: 72.07 -> 79.42 | Helped |
| replace ZIP with SharpZO | CLIP · EuroSAT time-to-test-accuracy | minutes: 251.4 -> 12.7 | Helped |
| add sharpness-aware CMA-ES warm-up instead of naive CMA-ES (ablation) | SharpZO · EuroSAT, ViT-B/16, Stage 1 and subsequent Stage 2 | accuracy and convergence | Helped |
| add Z-pruning instead of dense ZO training (ablation) | SharpZO · EuroSAT, ViT-B/16 | accuracy and convergence speed | Helped |
| use Z-Score pruning instead of magnitude pruning (ablation) | SharpZO · EuroSAT, ViT-B/16, sparsity 50% | accuracy: 76.12 -> 79.42 | Helped |
| set sharpness scaling factor rho to 0.5 instead of 0.1 (ablation) | SharpZO · EuroSAT, ViT-B/16 | accuracy: 79.42 -> 76.32 | Hurt |
| set sharpness scaling factor rho to 0.001 instead of 0.1 (ablation) | SharpZO · EuroSAT, ViT-B/16 | accuracy: 79.42 -> 77.74 | Hurt |
| vary Z-Score pruning sparsity from 50% to 10%, 30%, 70%, or 90% (ablation) | SharpZO · EuroSAT, ViT-B/16 | accuracy: 79.42; 78.91; 79.02; 79.15; 79.21 | Mixed |
Surprising findings
- Magnitude-based pruning performs worse than dense training for prompt tuning because the trainable parameter count (512) is already small, and the paper argues a sparsity ratio below 0.5 hurts prompt capacity, contrary to prior ZO full-model fine-tuning work.
Limitations
- Stage 1 needs coordinate-wise gradient estimation whose cost scales with the parameter dimension, limiting use beyond low-dimensional prompt tuning.
- Scalability to full-model or multimodal fine-tuning is unexplored.
- The convergence theorem relies on a Polyak-Lojasiewicz assumption and uses a simplified CMA-ES-as-natural-gradient argument with covariance dynamics omitted.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT DTD SUN397 UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft EuroSAT DTD SUN397 UCF101 ImageNetV2
Metrics: accuracy average accuracy time-to-test-accuracy (minutes) peak training memory (MB)
Baselines: Zero-shot CLIP BBT BlackVIP ZIP CraFT CoOp
Builds on: CMA-ES Sharpness-Aware Minimization BlackVIP ZIP BBT
Closest papers by meaning
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.90)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.89)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.89)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.89)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.89)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.