Research map / Architectures and efficiency / Pruning and efficient tuning
Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaPEFT, which meta-learns a differentiable per-position scalar modulator controlling insertion position, layer depth, and scaling of additive PEFT modules (LoRA, Adapter, AdaptFormer) via bi-level optimization for long-tailed remote sensing adaptation.
Problem
Additive PEFT methods for adapting natural-image foundation models to long-tailed remote sensing data are highly sensitive to hyperparameters (intra-block position, block depth, scaling factor), and jointly searching these mixed discrete-continuous settings is combinatorially expensive.
Approach
Unify discrete insertion positions and continuous scaling factors into one differentiable non-negative scalar modulator per PEFT position, and optimize it in the outer loop of a bi-level scheme on a random 20% held-out split of training data, while PEFT weights are updated in the inner loop.
Key results
- With LoRA, MetaPEFT raises average accuracy on IN21K→DOTA to 93.9 head/95.1 med/91.4 tail and overall average from 76.78 to 77.91 across three transfer scenarios.
- On SatMAE→SAR, LoRA with MetaPEFT improves tail-class accuracy from 72.1% to 74.2%.
- Adding the modulator to Adapter, AdaptFormer and LoRA yields consistent gains, with LoRA + MetaPEFT achieving the highest tail-class average (81.63%) in Table 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MetaPEFT meta-learning of insertion positions and scaling factors to LoRA | LoRA on IN21K ViT-B/16 · iNaturalist-2018 tail classes | accuracy: 78.5 -> 79.3 | Helped |
| Add MetaPEFT meta-learning of insertion positions and scaling factors to LoRA | LoRA on IN21K ViT-B/16 · DOTA tail classes | accuracy: 90.7 -> 91.4 | Helped |
| Add MetaPEFT meta-learning of insertion positions and scaling factors to LoRA | LoRA on SatMAE-L/16 · FUSRS v2 (SAR) tail classes | accuracy: 72.1 -> 74.2 | Helped |
| Add LoRA in both attention and feedforward layers rather than attention layers only (ablation) | LoRA on IN21K ViT-B/16 · DOTA tail classes | accuracy: 91.1 -> 92.4 | Helped |
| Insert LoRA at MLP 1 rather than at K (ablation) | LoRA on IN21K ViT-B/16 · DOTA | accuracy (average): 90.6 -> 93.4 | Helped |
| Apply LoRA to middle-lower blocks L3-5 rather than deepest blocks L9-11 (ablation) | LoRA on IN21K ViT-B/16 · DOTA | accuracy (average): 89.0 -> 91.9 | Helped |
| Increase outer-loop sampling ratio from 10% to 30% (ablation) | MetaPEFT · DOTA | accuracy (average): 92.8 -> 94.7 | Helped |
| Increase outer-loop sampling ratio from 10% to 30% (ablation) | MetaPEFT · DOTA tail classes | accuracy: 90.8% to 93.4% | Helped |
Surprising findings
- Combining individually optimal hyperparameters (e.g., best intra-block position with best block depth) can degrade accuracy, showing non-monotonic interactions.
- Scaling factor alone can swing accuracy from about 91% to 8% on the K projection layer, indicating extreme sensitivity.
Limitations
- Gains are modest on some settings (e.g., AdaptFormer shows no improvement on average), and differences are around 1 percentage point.
- Outer-loop sampling ratio trades performance against training time; 20% default chosen for time constraints.
- The method has many dataset-specific hyperparameters (meta learning rate, inner steps, batch sizes) that vary per dataset and backbone.
Experimental setup
Datasets: CIFAR100-IR100 Places-LT iNaturalist-2018 DOTA FUSRS v2 FUSAR-Ship SRSDD
Benchmarks: CIFAR100-IR100 Places-LT iNaturalist-2018 DOTA FUSRS v2
Metrics: accuracy head/medium/tail accuracy inter-class cosine distance
Baselines: VPT-Shallow VPT-Deep BitFit Adapter AdaptFormer LoRA
Builds on: LoRA AdaptFormer Adapter MAML Logit Adjustment loss
Closest papers by meaning
- Parameter Efficient Self-Supervised Geospatial Domain Adaptation (CVPR 2024 · similarity 0.87)
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · similarity 0.86)
- A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening (CVPR 2025 · similarity 0.86)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.85)
- OTPNet: ODE-inspired Tuning-free Proximal Network for Remote Sensing Image Fusion (AAAI 2025 · similarity 0.85)
- SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.