Research map / Large language models / Instruction and fine-tuning
Activated LoRA: Fine-tuned LLMs for Intrinsics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Activated LoRA (aLoRA), which adapts weights only for tokens after an invocation sequence so the base model's KV cache can be reused, enabling efficient multi-turn "intrinsics" with accuracy comparable to LoRA.
Problem
Switching between LoRA adapters in multiturn chains requires recomputing the entire KV cache of prior context with the adapter weights, causing large latency and memory overhead.
Approach
Adapt attention Q/K/V weights only for tokens at and after an invocation sequence, so keys and values for earlier tokens match the base model exactly and the base model's KV cache can be reused by the adapter (and vice versa).
Key results
- Across 1120 SFT adapter training runs on 7 tasks and 4 models, mean accuracy difference between LoRA and aLoRA was 0.6% in favor of LoRA (median 0.0%, p=0.8).
- On vLLM, aLoRA evaluator generation speedups reach up to 35x over LoRA depending on base model and prompt length, with over 20x for most models on the longest prompts.
- aLoRA intrinsics (uncertainty, answerability, query rewrite, jailbreak) show performance within about a point or two of corresponding LoRA adapters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reuse the base model KV cache by activating adapter weights only after invocation | aLoRA evaluator adapters vs LoRA · vLLM evaluator generation with varied prompt lengths | inference speed: over 2 to 7$\times$ even for 250-token prompts; over 20$\times$ for most models on the longest prompts | Helped |
| Reuse the base model KV cache by activating adapter weights only after invocation | aLoRA evaluator adapters vs LoRA · vLLM evaluator generation | inference speed: up to 35x improvement | Helped |
| Use aLoRA instead of standard LoRA | benchmark SFT adapters · 7 benchmark SFT tasks across 4 models | accuracy: median performance difference is 0.0\%; mean performance difference is 0.6\% in favor of LoRA; p-value ... is 0.8 | No clear effect |
| Use aLoRA instead of standard LoRA | benchmark SFT adapters · multiple-choice benchmark SFT tasks | accuracy: 3.06\% in favor of activated LoRA | Mixed |
| Use aLoRA instead of standard LoRA | Uncertainty Quantification intrinsic · Uncertainty Quantification test set | mean absolute error | No clear effect |
| Use aLoRA instead of standard LoRA | Answerability Determination intrinsic · SQUADRun Dev and MT-RAG | classification performance | No clear effect |
| Use aLoRA instead of standard LoRA | Query Rewrite intrinsic · MT-RAG | retrieval recall | No clear effect |
| Use aLoRA instead of standard LoRA | Query Rewrite intrinsic · MT-RAG | answer quality | No clear effect |
| Use aLoRA instead of standard LoRA | Jailbreak Detection intrinsic · out-of-distribution jailbreak datasets | classification performance | No clear effect |
Surprising findings
- aLoRA sometimes requires higher rank than LoRA to achieve good performance, though rank increase has negligible inference cost.
Limitations
- aLoRA-generated KV cache cannot be reused by other adapters or the base model, requiring re-prefill.
- aLoRA sometimes needs higher rank than LoRA for good performance, increasing training cost.
- Accuracy differences per task are noisy due to small datasets; only one base model used for intrinsics.
Experimental setup
Datasets: Super-Natural Instructions tasks via Lots-of-LoRAs SQUADRun MT-RAG BigBench MRQA trivia_qa super_glue ag_news jailbreak detection datasets
Benchmarks: Lots-of-LoRAs SFT tasks (7 tasks) SQUADRun dev MT-RAG Uncertainty quantification test set jailbreak out-of-distribution set (3,282 samples)
Metrics: accuracy mean absolute error precision recall F1 Recall@k RAGAS Faithfulness RAD-Bench score latency speedup time to first token
Baselines: LoRA
Builds on: LoRA vLLM prefix tuning QLoRA DoRA
Closest papers by meaning
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.89)
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.89)
- $\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning (NeurIPS 2024 · similarity 0.89)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.89)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.89)
- LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.