Research map / Architectures and efficiency / Pruning and efficient tuning
$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Trans-LoRA, a data-free method that transfers LoRA/PEFT modules between base LLMs by distilling on synthesized data filtered by a meta model trained to separate real from synthetic questions.
Problem
PEFT modules such as LoRA are tied to the specific base model they were trained on, so they must be retrained when the base model is deprecated or replaced, and the original training data may be unavailable.
Approach
Synthesize task-like data with few-shot LLM prompting, then keep only samples that a meta model (trained to distinguish real from synthetic questions, alongside the PEFT) judges as real, and distill the source PEFT+base model into a new PEFT on the target base model.
Key results
- On BBH, Llama-2-7b to Llama-2-13b transfer reached 54.48 average accuracy versus 56.38 for the finetuned source on tasks where the target outperforms the base.
- On MMLU, Llama-2-7b to Llama-2-13b transfer reached 62.46 versus 50.14 for the finetuned source on tasks where the target outperforms the base.
- Ablation on BBH: random Wikipedia distillation scored 25.33 and unfiltered synthetic data 45.62, versus 54.48 for the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer the LoRA from Llama-2-7b to Llama-2-13b using filtered synthetic data | Trans-LoRA · BigBench-Hard tasks where target outperforms base | accuracy: 56.38 -> 54.48 | No clear effect |
| Transfer the LoRA from Gemma-2b to Gemma-7b using filtered synthetic data | Trans-LoRA · BigBench-Hard tasks where target outperforms base | accuracy: 56.23 -> 54.16 | No clear effect |
| Transfer the LoRA from Llama-2-7b to Gemma-7b using filtered synthetic data | Trans-LoRA · BigBench-Hard tasks where target outperforms base | accuracy: 52.98 -> 57.21 | Helped |
| Transfer the LoRA from Llama-2-7b to Llama-2-13b using filtered synthetic data | Trans-LoRA · MMLU tasks where target outperforms base | accuracy: 50.14 -> 62.46 | Helped |
| Transfer the LoRA from Gemma-2b to Gemma-7b using filtered synthetic data | Trans-LoRA · MMLU tasks where target outperforms base | accuracy: 38.81 -> 61.34 | Helped |
| Use random Wikipedia text instead of filtered synthetic data for distillation (ablation) | Trans-LoRA on Llama-2-13b · BigBench-Hard tasks where target outperforms Llama-2-7b | accuracy: 54.48 -> 25.33 | Hurt |
| Use unfiltered synthetic data instead of meta-model-filtered data for distillation (ablation) | Trans-LoRA on Llama-2-13b · BigBench-Hard tasks where target outperforms Llama-2-7b | accuracy: 54.48 -> 45.62 | Hurt |
Surprising findings
- A meta model trained to predict whether the teacher answers a question correctly reaches high accuracy but does not improve transfer, so teacher accuracy on the data is not the key factor.
Limitations
- Experiments restricted to LoRA and to tasks where the target model outperforms the base model, and some tasks fall below baseline (e.g., disambiguation-qa due to invalid synthetic questions).
- Only one meta-model architecture was used; its design effect was not explored.
- Evaluation covers only BBH and MMLU with a small set of model pairs.
Experimental setup
Datasets: BigBench-Hard MMLU WikiText (random text ablation)
Benchmarks: BigBench-Hard MMLU
Metrics: accuracy
Baselines: Random Wikipedia text distillation Unfiltered synthetic data distillation Finetuned weaker base model with LoRA
Builds on: LoRA Weak-to-strong generalization Knowledge distillation
Closest papers by meaning
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
- AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping (NeurIPS 2025 · similarity 0.91)
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.