Research map / Large language models / Instruction and fine-tuning
LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA-Gen, where a cloud-side LLM (LLaMA3-8B) generates layer-wise LoRA weights from task system prompts via meta tokens and a LoRA expert pool, which are merged into an edge-side small LM to specialize it without per-task training.
Problem
Specializing small edge-side LLMs to domain tasks with LoRA fine-tuning risks catastrophic forgetting and generalizes poorly to unseen tasks, while LoRA-MoE adds inference cost and long system prompts inflate context length.
Approach
A large cloud LLM encodes the task system prompt into per-layer meta tokens; a routing module composes a pool of LoRA experts into layer-specific LoRA weights that are reparameterized into the edge model, giving context compression and zero-training specialization for unseen tasks, with knowledge transfer from the cloud model.
Key results
- With TinyLLaMA-1.1B, LoRA-Gen reaches 55.1 AVE and 49.8 HAR versus 54.3/48.5 for LoRA, at 21.2 ms vs 44.5 ms latency (about 2.1x speedup).
- On GPT4Tools with Gemma-2B, LoRA-Gen without tool definitions in the prompt achieves 84.2 average score with a 10.1x compression ratio.
- Compared with AutoCompressors on OPT-2.7B unseen tasks, LoRA-Gen averages 61.6 vs 60.1 with lower latency (7.54 ms vs 11.4 ms).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoRA-Gen instead of LoRAMoE | TinyLLaMA-1.1B · Commonsense reasoning, 5-shot | HAR: 49.3 -> 49.8 | Helped |
| Use LoRA-Gen instead of LoRA | TinyLLaMA-1.1B · ARC-c latency | Latency (ms): 44.5 -> 21.2 | Helped |
| Use LoRA-Gen instead of MixLoRA | Qwen-1.5B · Commonsense reasoning, 5-shot | HAR: 54.2 -> 55.0 | Helped |
| Use LoRA-Gen instead of LoRA | Qwen-1.5B · ARC-c latency | Latency (ms): 56.3 -> 26.7 | Helped |
| Use LoRA-Gen instead of MixLoRA | Gemma-2B · Commonsense reasoning, 5-shot | HAR: 60.2 -> 60.2 | No clear effect |
| Use LoRA-Gen instead of LoRA | Gemma-2B · ARC-c latency | Latency (ms): 87.3 -> 36.1 | Helped |
| Specialize without GPT4Tools training or tool definitions | Gemma-2B with LoRA-Gen · GPT4Tools test set | Average score: 79.3 -> 84.2 | Helped |
| Use LoRA-Gen instead of AutoCompressors | OPT-2.7B · HellaS, WinoG, PIQA unseen tasks | AVE.: 60.1 -> 61.6 | Helped |
| Use 4 experts instead of 8 (ablation) | LoRA-Gen · Expert-count ablation | AVE.: 58.7 -> 56.4 | Hurt |
| Use 12 experts instead of 8 (ablation) | LoRA-Gen · Expert-count ablation | AVE.: 58.7 -> 57.3 | Hurt |
| Set auxiliary-loss coefficient to 0.1 instead of 0.01 (ablation) | LoRA-Gen · Auxiliary-loss coefficient ablation | AVE.: 58.7 -> 57.1 | Hurt |
| Set auxiliary-loss coefficient to 0.005 instead of 0.01 (ablation) | LoRA-Gen · Auxiliary-loss coefficient ablation | AVE.: 58.7 -> 56.8 | Hurt |
Surprising findings
- LoRA-Gen with only 1-shot samples surpasses the 5-shot baseline by 3.5% HAR, which the authors attribute to knowledge transfer from the LLaMA3-8B cloud model.
- Directly generating LoRA weights without meta tokens matches seen-task performance but drops 11.1 points on unseen tasks.
Limitations
- Generated LoRA training requires large cloud-side model and a modified Alpaca dataset via GPT-4, adding setup cost.
- Gumbel-softmax routing reduced generalization, and generating LoRA directly overfits to training domains.
- Validation limited to LLMs; multimodal and unified systems left to future work.
Experimental setup
Datasets: ARC-c ARC-e OpenBookQA BoolQ SocialIQA HellaSwag WinoGrande PIQA GPT4Tools Alpaca
Benchmarks: ARC-c ARC-e OpenBookQA BoolQ SocialIQA HellaSwag WinoGrande PIQA GPT4Tools
Metrics: accuracy average accuracy harmonic mean latency success rate IoU compression ratio
Baselines: LoRA LoRAMoE MixLoRA AutoCompressors LoraHub ICL
Builds on: LoRA LoRA-MoE LoraHub AutoCompressors Gumbel-Softmax
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.89)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.89)
- HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · similarity 0.89)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.