Research map / Architectures and efficiency / Pruning and efficient tuning
LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation
Hao Huang, Jingtao Ding, Mengqi Liao, Xin Wang, Jinyang Ban, Yuan Yuan, Huaiyu Wan, Yong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add module-aware MoE decoder (ablation) | LoRAGen · FLAN subset (in-distribution) | average accuracy: 58.4 -> 96.0 | Improved |
| add direction and spectral adapter-level losses to the MoE decoder (ablation) | LoRAGen · FLAN subset (in-distribution) | average accuracy: 95.2 -> 96.0 | Improved |
| add direction loss alongside the spectral loss (ablation) | LoRAGen · FLAN subset (in-distribution) | average accuracy: 36.9 -> 96.0 | Improved |
| add adapter-level losses to the decoder for unseen tasks (ablation) | LoRAGen · FLAN subset (zero-shot) | average accuracy: 36.6 -> 40.2 | Improved |
| use LoRAGen instead of T2L on unseen tasks | LoRAGen · FLAN subset (zero-shot) | average accuracy: 35.2 -> 40.2 | Improved |
| use LoRAGen instead of T2L on in-distribution tasks | LoRAGen · FLAN subset (in-distribution) | average accuracy: 88.7 -> 96.0 | Improved |
| use LoRAGen vs task-specific LoRAs on Gemma-2-2B HellaSwag | LoRAGen · HellaSwag (Gemma-2-2B-Instruct) | accuracy: 75.4 -> 64.1 | Hurt |
| use LoRAGen vs task-specific LoRAs on Gemma-2-2B average | LoRAGen · 8 benchmark tasks (Gemma-2-2B-Instruct) | average accuracy: 74.5 -> 72.7 | Hurt |
| add structural embedding to the MoE decoder routing (ablation) | LoRAGen · FLAN subset (in-distribution) | accuracy | Improved |
| use unshared expert pools instead of shared pools (ablation) | LoRAGen MoE decoder · FLAN subset (in-distribution) | accuracy | Improved |
| increase active experts from top-1 to top-2 (ablation) | LoRAGen MoE decoder · FLAN subset (in-distribution) | accuracy | Improved |
| vary spectral-energy threshold rho from 0.80 to 1.00 (ablation) | LoRAGen spectral loss · FLAN subset (in-distribution) | accuracy | No clear effect |
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.91)
- Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment (ICML 2025 · similarity 0.91)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.91)
- Text-to-LoRA: Instant Transformer Adaption (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
