Research map / Large language models / Instruction and fine-tuning
Text-to-LoRA: Instant Transformer Adaption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Text-to-LoRA (T2L), a hypernetwork that generates task-specific LoRA adapters in one forward pass from a natural-language task description, trained either by distilling pre-trained LoRAs or by multi-task SFT.
Problem
Adapting LLMs to new tasks requires per-task dataset curation and LoRA fine-tuning with sensitive hyperparameters; existing LoRA compression and routing methods do not generate adapters zero-shot from natural language.
Approach
A hypernetwork conditioned on text embeddings of task descriptions plus learned module and layer embeddings outputs LoRA A/B matrices for every target module in a single batched forward pass; it can be trained by reconstructing pre-trained LoRAs or end-to-end via SFT on many tasks, enabling zero-shot adapter generation for unseen tasks.
Key results
- Reconstruction-trained T2L on 9 benchmark LoRAs reaches avg 73.4-73.5 vs 73.3 for task-specific LoRAs and 55.8 for the base model.
- SFT-trained T2L (Larch) generates zero-shot LoRAs averaging 67.7 over 10 benchmarks, above multi-task LoRA (66.3) and 3-shot ICL (61.0), but below Hyperdecoders (67.3 avg on a different set) in the top-line comparison and below oracle task-specific LoRAs on most tasks.
- With Llama-3.1-8B-Instruct, T2L (SFT) Larch averages 76.9 vs 76.5 for multi-task LoRA; with Gemma-2-2B-Instruct it averages 66.4 vs 65.2 for multi-task LoRA w/ ICL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the hypernetwork with SFT rather than LoRA reconstruction (ablation) | T2L · 10 benchmark tasks | average benchmark performance: 61.8 -> 66.3 | Helped |
| use task-aligned descriptions rather than descriptions from unrelated tasks (ablation) | reconstruction-trained T2L · 9 benchmark tasks | average benchmark performance: 51.4 -> 73.3 | Helped |
| use generated descriptions instead of SNI-provided definitions (ablation) | T2L SFT Larch · 10 benchmark tasks | average accuracy/pass@1: 66.5 -> 67.7 | Helped |
| increase the number of SFT training tasks with proportionally scaled compute (ablation) | T2L · 10 benchmark tasks | average accuracy/pass@1: 65.2 -> 67.7 | Mixed |
| use Mistral rather than gte task-description embeddings (ablation) | T2L SFT · 10 benchmark tasks | average benchmark performance: 65.9 -> 65.6 | No clear effect |
| use more training tasks without increasing the compute budget (ablation) | T2L SFT · 10 benchmark tasks | average accuracy/pass@1: Larch: 66.0, 66.6, 67.5, 66.6 | Mixed |
| generate LoRAs from task descriptions with SFT | T2L · 10 benchmark tasks | average accuracy/pass@1: 66.3 -> 67.5 | Helped |
| compress task-specific LoRAs with reconstruction-trained T2L | T2L · 9 benchmark tasks | average accuracy/pass@1: 73.3 -> 73.5 | No clear effect |
| use T2L SFT Larch rather than multi-task LoRA | Llama-3.1-8B-Instruct · 10 benchmark tasks | average accuracy/pass@1: 76.5 -> 76.9 | Helped |
| use T2L SFT Larch with ICL rather than multi-task LoRA with ICL | Gemma-2-2B-Instruct · 10 benchmark tasks | average accuracy/pass@1: 65.2 -> 66.4 | Helped |
Surprising findings
- Reconstruction-trained T2L sometimes outperforms the task-specific oracle LoRAs (e.g., on PIQA and WG), which is attributed to lossy compression acting as regularization.
- Reconstruction-trained T2L fails to generalize to unseen tasks, since LoRAs for semantically similar tasks are not nearby in weight space (near-zero correlation between adapter similarity and description similarity).
Limitations
- Zero-shot T2L does not reach task-specific LoRA performance on benchmarks.
- Performance is sensitive to the quality and alignment of task descriptions; unaligned descriptions degrade adapters.
- Only LoRA is used as the hypernetwork output space; the authors note more efficient modulation methods may exist.
Experimental setup
Datasets: Super-Natural Instructions (SNI) via Lots-of-LoRAs ARC-Challenge ARC-Easy BoolQ GSM8K HellaSwag OpenBookQA PIQA Winogrande HumanEval MBPP
Benchmarks: ARC-Challenge ARC-Easy BoolQ GSM8K HellaSwag OpenBookQA PIQA Winogrande HumanEval MBPP
Metrics: accuracy pass@1 average benchmark performance
Baselines: Task-specific LoRA Multi-task LoRA Average LoRA 3-shot ICL Prepending task description Arrow Routing Hyperdecoders
Builds on: LoRA Hyperdecoders Arrow Routing Super-Natural Instructions
Closest papers by meaning
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.91)
- LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · similarity 0.90)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.90)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.89)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.89)
- Doc-to-LoRA: Learning to Instantly Internalize Contexts (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.