Research map / Large language models / Instruction and fine-tuning
LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA Recycle, which distills a meta-LoRA from many pre-tuned LoRAs using synthetic data inverted from those LoRAs, enabling tuning-free few-shot classification in CLIP ViT-B/16 with token pruning for efficiency.
Problem
Visual foundation models lack the tuning-free few-shot adaptability of LLMs and fine-tuning with very few examples is unstable and slow, while reusing pre-tuned LoRAs is constrained by unavailable original training data.
Approach
Invert each pre-tuned LoRA into synthetic data (LoRA Inversion with BN-statistics regularization), then meta-train a meta-LoRA with a prototype-based (Euclidean class-center) inner loop and KL distillation from the pre-tuned LoRA teachers, so the VFM adapts to new few-shot tasks in a single forward pass; token pruning accelerates both inversion and meta-training.
Key results
- LoRA Recycle reaches 89.69% 5-way 1-shot on CIFAR-FS and outperforms the best fine-tuning baseline (LoRAHub 81.23%) by up to 9.80% in 1-shot in-domain settings.
- In cross-domain settings, LoRA Recycle exceeds the best fine-tuning-based baselines by up to 4.31% (1-shot) and 2.41% (5-shot).
- Pruning 75% of synthetic tokens speeds meta-training up to 3x (throughput 6.83 vs 1.76 its/s on CUB) while keeping accuracy comparable or slightly higher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Recycle pre-tuned LoRAs to distill a meta-LoRA for tuning-free few-shot adaptation | ViT-B/16 CLIP · CIFAR-FS 5-way 1-shot | accuracy: 81.23 -> 89.69 | Helped |
| Recycle pre-tuned LoRAs to distill a meta-LoRA for tuning-free few-shot adaptation | ViT-B/16 CLIP · ISIC 5-way 5-shot | accuracy: 46.12 -> 47.96 | Helped |
| Recycle pre-tuned LoRAs to distill a meta-LoRA for tuning-free few-shot adaptation | ViT-B/16 CLIP · EuroSAT 5-way 1-shot and 5-way 5-shot | accuracy: 69.45 -> 66.95; 89.76 -> 85.17 | Hurt |
| Use token-masked synthetic images for meta-training (ablation) | LoRA Recycle on CIFAR-FS 5-way 1-shot · CIFAR-FS | accuracy: 89.69 -> 91.03; 89.69 -> 90.91; 89.69 -> 89.70 | Mixed |
| Use token-masked synthetic images for meta-training (ablation) | LoRA Recycle on CUB 5-way 1-shot · CUB | accuracy: 91.12 -> 90.16; 91.12 -> 90.65; 91.12 -> 91.21 | Mixed |
| Use 75% sparse tokens for meta-training (ablation) | LoRA Recycle on CUB · CUB meta-training | throughput: 1.76 -> 6.83 (+287%) | Helped |
| Prune 75% of tokens at layer 11 during inversion (ablation) | LoRA Recycle on CIFAR-FS · CIFAR-FS 5-way few-shot | throughput: 5.56 -> 5.81 (+4%) | Helped |
| Prune 75% of tokens at layer 6 during inversion (ablation) | LoRA Recycle on CIFAR-FS · CIFAR-FS 5-way 1-shot | accuracy: 89.69 -> 81.08 | Hurt |
| Prune 75% of tokens at layer 8 during inversion (ablation) | LoRA Recycle on CIFAR-FS · CIFAR-FS inversion | throughput: 5.56 -> 6.22 (+12%) | Helped |
| Prune 75% of tokens at layer 8 during inversion (ablation) | LoRA Recycle on CIFAR-FS · CIFAR-FS 5-way 1-shot | accuracy: 89.69 -> 82.27 | Hurt |
Surprising findings
- Meta-training on sparse (token-pruned) synthetic data can improve accuracy by up to 1.34%, attributed to removing noisy background tokens.
- Naive LoRA averaging and LoRAHub do not generalize well to new tasks, since averaging in parameter space lacks alignment across different label spaces.
Limitations
- Performance gains degrade under substantial distribution shift in cross-domain scenarios.
- Evaluated only on few-shot image classification with CLIP ViT backbones and 5-way tasks.
Experimental setup
Datasets: CIFAR-FS MiniImageNet VGG-Flower CUB ChestX ISIC EuroSAT CropDiseases
Benchmarks: CIFAR-FS MiniImageNet VGG-Flower CUB ChestX ISIC EuroSAT CropDiseases
Metrics: accuracy on 600 5-way 1-shot and 5-way 5-shot tasks throughput GPU memory FLOPs
Baselines: Full Finetuning Linear probe LoRA + Linear P>M>F LoRAs Avg + Linear MOLE LoRAHub NN LoRAs Avg + NN CAML
Builds on: LoRA LoRAHub Data-Free Meta-Learning CAML P>M>F
Closest papers by meaning
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.91)
- Text-to-LoRA: Instant Transformer Adaption (ICML 2025 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.90)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.90)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.90)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.