Research map / Label-efficient and robust learning / Recognition and long-tail

PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

NeurIPS 2024 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PACE, a PEFT method that perturbs adapter features with multiplicative Gaussian noise and enforces consistency between two noised forward passes, theoretically linking this to smaller weight gradients and alignment with the pre-trained model.

Problem

PEFT adaptation of pre-trained transformers often sacrifices generalization; prior work lacks a theoretical link between gradient norms and generalization and naive alignment to the pre-trained model can cause gradient explosion.

Approach

Perturb adapter outputs with multiplicative noise z~N(1,σ²) and penalize output differences between two noised copies of the network; theory shows this implicitly penalizes first- and second-order weight gradients and reduces the FP-distance to the pre-trained model, unlike direct FP-distance alignment.

Key results

• On VTAB-1K with ViT-B/16, PACE raises the LoRAmul+VPTadd baseline mean accuracy from 76.4 to 79.0, slightly above GLoRA (78.0).
• On GSM-8K with Phi-3-mini, LoRA 75.66 vs +PACE 78.77 accuracy.
• On GLUE with RoBERTa-base, PACE improves LoRA average from 86.6 to 87.6.

Measured effects reported in the paper

Surprising findings

• Naive fine-tuned/pre-trained alignment (FPA) can cause gradient explosion for some regularization strengths, while PACE reliably lowers gradient norms.
• PACE's storage-light variant PACEfast with 1/8 batch size and epochs still outperforms the baseline on CIFAR-100, Camelyon and ImageNet while using about one third of GPU memory.

Limitations

• Training requires passing inputs through the network twice, increasing computational and memory cost.
• Introduces extra hyperparameters λ and σ that require grid search and careful tuning.
• Theoretical analysis uses a one-dimensional output and second-order Taylor approximation, assuming small adapter updates.

Experimental setup

Datasets: VTAB-1K FGVC-Aircraft Food101 Oxford Flowers102 Oxford Pets Stanford Cars CUB-200-2011 NABirds Stanford Dogs ImageNet-1K GLUE GSM-8K

Benchmarks: VTAB-1K few-shot learning FGVC domain adaptation (ImageNet-Sketch, V2, A, R) GLUE GSM-8K

Metrics: accuracy Matthews correlation Pearson correlation gradient norm FP-distance

Baselines: GLoRA RLRR RepAdapter NOAH AdaptFormer SSF ARC VeRA BitFit LoRA L2SP DELTA

Builds on: LoRA GLoRA VPT R-Drop NICE sharpness-aware minimization

Closest papers by meaning

• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.88)
• Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.88)
• LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.88)
• Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.88)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.88)