Research map / Architectures and efficiency / Pruning and efficient tuning

Parameter-Efficient Fine-Tuning with Controls

ICML 2024 · Architectures and efficiency › Pruning and efficient tuning · approach: LoRA variants and theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Recasts LoRA-style PEFT as a control problem over frozen ViT blocks, derives a controllability condition, and proposes a parameter-free cross-patch attention controller (ANC) that beats LoRA and AdaptFormer on three vision datasets at equal parameter counts.

Problem

Parameter-efficient fine-tuning of pre-trained Vision Transformers lacks a principled understanding; existing LoRA-like perturbations (LoRA, AdaptFormer) may fail to be controllable, especially for large datasets, and token-wise linear controllers cannot mix information across patches.

Approach

Treat lightweight LoRA matrices as control modules that perturb frozen ViT dynamics; derive sufficient conditions for local controllability (linear controllers fail when datasets are large), and add a parameter-free cross-patch attention head on top of LoRA outputs to introduce nonlinearity and token mixing without extra parameters.

Key results

• ANC-32 reaches 88.06% on Food-101 vs 85.70% for LoRA-16 and 86.21% for AdaptFormer-32, at the same 0.67M parameters.
• ANC-128-2H reaches 87.52% on CIFAR-100, close to Full-Tuning's 87.90% while tuning 4.82% of parameters.
• Replacing ReLU in AdaptFormer with a sigmoid lowers CIFAR-100 accuracy by about 1.4 points (84.61% vs 85.90%).

Measured effects reported in the paper

Surprising findings

• Removing the ReLU from AdaptFormer (linear control) performs about the same as the ReLU version, and a sigmoid nonlinearity hurts performance.
• Increasing nonlinearity via a 2-head controller gives a slightly larger gain than doubling LoRA rank at equal parameter count (ANC-32-2H vs ANC-64).

Limitations

• Empirical validation limited to ViT on three image classification datasets with a single run for most results.
• Controllability theory gives only sufficient conditions; the nonlinear controller's satisfaction is shown numerically on a toy example, not proven in general.
• The multi-head variant adds parameters, so the parameter-free claim applies only to the single-head version.

Experimental setup

Datasets: CIFAR-100 SVHN Food-101

Benchmarks: CIFAR-100 SVHN Food-101

Metrics: test accuracy training loss singular value spectrum (controllability check)

Baselines: Full-Tuning Linear-Probing VPT LoRA AdaptFormer

Builds on: LoRA AdaptFormer Vision Transformer Visual Prompt Tuning

Closest papers by meaning

• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.89)
• LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.89)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation (NeurIPS 2026 · similarity 0.89)
• The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.89)
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.89)
• Asymmetry in Low-Rank Adapters of Foundation Models (ICML 2024 · similarity 0.89)