Research map / Architectures and efficiency / Pruning and efficient tuning
Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPNs, a data-model co-design that jointly learns visual prompts (pad prompts) and parameterized weight masks to find sparse subnetworks that outperform SoTA pruning on ImageNet-pretrained CNNs across eight downstream datasets.
Problem
Conventional pruning is model-centric and post-pruning visual prompts do not help sparse vision models; the interplay of data (visual prompts) and pruning is unexplored.
Approach
Co-optimize a learnable visual pad prompt with a learned binary weight mask (score-based, HYDRA-style) during mask finding, then jointly fine-tune weights and prompt with the mask fixed, so the input data design is used to locate better sparse subnetworks.
Key results
- At 90% sparsity with ResNet-18 on Tiny-ImageNet, VPNs outperforms HYDRA, BiP, and LTH by 3.41%, 1.69%, and 2.00%.
- VPNs with ResNet-18 at 10% and 20% channel sparsity achieves 1.1x and 1.3x speedups while reducing memory by 15.26% and 31.71% without accuracy loss versus dense.
- For 90% sparsity, VPNs needs 95%, 50%, and 50% fewer epochs than LTH, GraSP, and HYDRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VPNs instead of HYDRA | ResNet-18 pruning · Tiny ImageNet, 90% sparsity | test accuracy: +3.41% | Helped |
| Use VPNs instead of BiP | ResNet-18 pruning · Tiny ImageNet, 90% sparsity | test accuracy: +1.69% | Helped |
| Use VPNs instead of LTH | ResNet-18 pruning · Tiny ImageNet, 90% sparsity | test accuracy: +2.00% | Helped |
| Use VPNs instead of OMP | ResNet-18 pruning · CIFAR100, 90% sparsity | test accuracy: +3.46% | Helped |
| Use VPNs instead of OMP | ResNet-18 pruning · Tiny ImageNet, 80% sparsity | test accuracy: +3.97% | Helped |
| Use VPNs instead of HYDRA | ResNet-18 pruning · CIFAR100, 90% sparsity | test accuracy: +2.75% | Helped |
| Combine the pruning method with VPNs | Random, OMP, and LTH pruning on ResNet-18 · CIFAR100, 80% sparsity | test accuracy: +1.16%, +0.81%, +0.79% | Helped |
| Use a larger input size, increasing from 128 toward 224 (ablation) | VPNs with pad size 16 · CIFAR100 | test accuracy | Helped |
| Increase pad size beyond 32 (ablation) | VPNs with input size 224 · CIFAR100 | test accuracy | Hurt |
| Use pad prompt rather than fix or random prompt (ablation) | VPNs · CIFAR100 | test accuracy | Helped |
| Add visual prompting during subnetwork tuning (ablation) | VPNs · CIFAR100 | test accuracy | Helped |
| Add visual prompting during both mask finding and subnetwork tuning (ablation) | VPNs · CIFAR100 | test accuracy | Helped |
Surprising findings
- Post-pruning visual prompts (zero-shot or with fine-tuning) do not consistently beat the standard prune-and-fine-tune baseline.
Limitations
- Gains over the best baselines on some datasets are modest (e.g., 0.8-2%).
- Evaluated only on CNN classification architectures (ResNet, VGG) and image classification tasks.
Experimental setup
Datasets: ImageNet-1K Tiny-ImageNet StanfordCars OxfordPets Food101 DTD Flowers102 CIFAR10 CIFAR100
Benchmarks: Tiny-ImageNet CIFAR100 CIFAR10 Food101 Flowers102 OxfordPets DTD StanfordCars
Metrics: test accuracy training time training epochs gradient steps FLOPs speedup memory cost
Baselines: Random pruning OMP LTH SNIP GraSP SynFlow HYDRA BiP Slim DepGraph
Builds on: HYDRA visual prompting (pad prompt) lottery ticket hypothesis SynFlow
Closest papers by meaning
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.89)
- LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation (ICLR 2025 · similarity 0.88)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.88)
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.86)
- MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning (CVPR 2024 · similarity 0.86)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.