Research map / Architectures and efficiency / Pruning and efficient tuning
Gradient-based Parameter Selection for Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPS, a parameter-efficient fine-tuning method that selects the top-K highest-gradient input connections per neuron (gradients from a supervised contrastive loss) and fine-tunes only those weights via a sparse mask, adding no parameters.
Problem
Full fine-tuning of large pre-trained vision models is memory-costly and data-inefficient, and existing PEFT methods add extra parameters or require architecture-specific designs that add training/inference cost.
Approach
Select a small, task-specific subset of pre-trained weights per neuron (top-K input connections by gradient modulus, computed with a head-free supervised contrastive loss) so the selection is distributed across the whole network and architecture-agnostic; then fine-tune only those weights with a binary gradient mask, introducing no extra parameters.
Key results
- GPS reaches 91.78% mean accuracy on FGVC with 0.77% tuned parameters, versus 88.54% for full fine-tuning and 90.72% for SSF.
- GPS reaches 75.18% mean accuracy on VTAB-1k with 0.25% parameters, versus 65.57% for full fine-tuning and 74.07% for SPT-LoRA.
- On polyp segmentation with SAM, GPS achieves 88.1 mDice and 72.5 mIoU, above SSF (87.3/71.7) and full fine-tuning (71.1/55.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPS parameter selection instead of full fine-tuning | ViT-B/16 · FGVC (mean over 5 tasks) | accuracy: 88.54 -> 91.78 | Helped |
| Use GPS parameter selection instead of SPT-LoRA | ViT-B/16 · VTAB-1k (mean over 19 tasks) | accuracy: 74.07 -> 75.18 | Helped |
| Use GPS to fine-tune SAM instead of SSF | SAM · Polyp segmentation | mDice: 87.3 -> 88.1 | Helped |
| Use GPS to fine-tune SAM instead of SSF | SAM · Polyp segmentation | mIoU: 71.7 -> 72.5 | Helped |
| Use GPS instead of full fine-tuning | Swin-B · FGVC (mean over 5 tasks) | accuracy: 92.42 -> 92.56 | Helped |
| Use GPS instead of full fine-tuning | ConvNeXt-B · FGVC (mean over 5 tasks) | accuracy: 93.04 -> 93.32 | Helped |
| Select the top-gradient connections per neuron instead of selecting highest-gradient parameters across the network or by layer (ablation) | GPS · FGVC | accuracy: 86.86 -> 87.17; 87.30 -> 88.07 | Helped |
| Use gradient-based rather than random parameter selection (ablation) | GPS · FGVC | accuracy: 86.60 -> 88.07; 87.17 -> 88.07 | Helped |
| Select neuron connections by gradient rather than weight magnitude (ablation) | GPS · FGVC | accuracy: 87.29 -> 88.07; 85.99 -> 86.64; 99.62 -> 99.69; 89.29 -> 90.10; 91.30 -> 92.30 | Mixed |
| Calculate selection gradients with supervised contrastive loss without a classification head rather than cross-entropy with a randomly initialized head (ablation) | GPS · FGVC | mean accuracy | Helped |
| Select parameters across the whole network rather than excluding parameters from transformer blocks (ablation) | GPS · CUB-2011 | accuracy | Hurt |
| Vary the number of selected input connections per neuron (ablation) | GPS · FGVC | Top-1 accuracy | Mixed |
Surprising findings
- Selecting only one input connection per neuron (about 0.25% of parameters) on VTAB-1k outperforms full fine-tuning by about 9.6 points in mean accuracy.
- Linear probing and full fine-tuning are weaker than GPS even on Swin-B and ConvNeXt-B, where SSF underperforms full fine-tuning.
Limitations
- More selected connections do not monotonically improve accuracy; the best count must be tuned per dataset.
- Selection relies on gradients from the downstream training data, adding a selection stage before fine-tuning.
- Gains on the ImageNet-1k and CIFAR-100 results are reported only in the supplement, so the main text covers a limited set of tasks.
Experimental setup
Datasets: CUB-200-2011 NABirds Oxford Flowers Stanford Dogs Stanford Cars VTAB-1k CIFAR-100 ImageNet-1k Polyp segmentation dataset
Benchmarks: FGVC VTAB-1k CIFAR-100 ImageNet-1k Polyp segmentation
Metrics: Top-1 accuracy mean Dice mean IoU number of tuned parameters (%) training/test time and memory
Baselines: Full fine-tuning Linear probing Bias tuning Adapter LoRA VPT-Shallow VPT-Deep SSF SPT-Adapter SPT-LoRA
Builds on: Visual Prompt Tuning SSF Adapter Supervised Contrastive Loss Child-Tuning
Closest papers by meaning
- Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation (AAAI 2025 · similarity 0.85)
- Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · similarity 0.84)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.84)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.84)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.84)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.