Research map / Architectures and efficiency / Pruning and efficient tuning

Provable Meta-Learning with Low-Rank Adaptations

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PEFT-based meta-learning (LoRA-ML) for retraining foundation models, proving standard retraining is suboptimal for LoRA adaptation in linear low-rank-perturbation regression and that LoRA-ML recovers ground truth for T≥3, with CIFAR-10 and ConvAI2 experiments.

Problem

Standard multi-task retraining of foundation models may not yield parameters that adapt well via parameter-efficient fine-tuning (LoRA) to unseen tasks; the benefits of PEFT-based meta-learning lack theoretical grounding.

Approach

Incorporate per-task LoRA adapters into the retraining objective (LoRA-ML) so base weights are optimized for post-adaptation loss across training tasks; in linear models with low-rank task perturbations of a shared matrix, this provably yields low-rank adaptable solutions and exact recovery when T≥3.

Key results

• On CIFAR-10 LoRA fine-tuning, LoRA-ML reaches 86.09% mean accuracy vs 85.29% for SR+LoRA and 81.30% for Reptile+LoRA.
• On ConvAI2 with 10 test tasks, LoRA-ML averages 47.4% accuracy vs 35.0% for SR+LoRA and 28.4% for Reptile+LoRA.
• Theory: for T≥3 and d≥3k, zero LoRA-ML loss implies exact recovery of the ground-truth shared matrix A*.

Measured effects reported in the paper

Surprising findings

• Standard retraining's fine-tuning error worsens as the number of retraining tasks T grows, since it inflates the needed adaptation rank to min{d,k(T+1)}.
• Reptile underperforms standard retraining because its gradient-based adaptation is misaligned with LoRA/last-layer fine-tuning.

Limitations

• Theory is restricted to linear models and symmetric low-rank adapters in the infinite-sample regime.
• Two-task landscape result only guarantees SOSP equals global minimum; spurious local minima exist for adversarial T≥3 instances.
• Experiments use simple models (MLP-Mixer, RoBERTa-base) and small-scale binary tasks.

Experimental setup

Datasets: CIFAR-10 ConvAI2

Benchmarks: CIFAR-10 (binary subsets) ConvAI2

Metrics: test accuracy squared prediction error

Baselines: Standard retraining (SR) Reptile Last-layer fine-tuning

Builds on: MAML Reptile LoRA

Closest papers by meaning

• Unraveling Model-Agnostic Meta-Learning via The Adaptation Learning Rate (ICLR 2022 · similarity 0.89)
• How Fine-Tuning Allows for Effective Meta-Learning (NeurIPS 2021 · similarity 0.88)
• Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.88)
• Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts (ICML 2024 · similarity 0.87)
• LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · similarity 0.87)
• Learning to Learn with Contrastive Meta-Objective (NeurIPS 2025 · similarity 0.87)