Research map / Architectures and efficiency / Pruning and efficient tuning
Batched Low-Rank Adaptation of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FLoRA, a LoRA variant that gives each example in a minibatch its own low-rank adapter so heterogeneous adapter requests can be batched, evaluated with throughput/latency serving tests and fine-tuning on MultiPL-E code generation and Whisper multilingual speech recognition.
Problem
LoRA adapters are shared across a whole minibatch, so serving requests that each need a different task- or user-specific adapter forces either a single shared adapter or sequential processing, hurting throughput and latency.
Approach
Factor each example's low-rank update as per-example B_i and A_i and rewrite the forward pass as elementwise-multiply, matrix-multiply with shared W0, and elementwise-multiply, so all per-example adapters are batched via matrix operations instead of torch.bmm, with cost analysis showing advantage at low rank.
Key results
- At rank 1 on StarCoder 15B, FLoRA throughput is more than 3x that of torch.bmm LoRA; at rank 2 it is about 2.5x.
- In the low-rank regime (rank 1 to 4) on StarCoder 15B at 8 requests/s, FLoRA gives a 2-5x latency reduction versus LoRA.
- On MultiPL-E low-resource languages, FLoRA reaches pass@1 comparable to LoRA and above IA3 (e.g., Ruby 24.94 vs LoRA 23.76 on StarCoder 15B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use example-specific adapters to batch heterogeneous requests | Fast LoRA · StarCoder serving, low ranks | throughput: more than threefold higher than that of LoRA | Helped |
| Use example-specific adapters to batch heterogeneous requests | Fast LoRA · StarCoder 15B serving, ranks 1–4 | latency: 2-5X reduction in latency compared to LoRA | Helped |
| Increase adapter rank (ablation) | Fast LoRA versus LoRA · StarCoder serving | throughput | Mixed |
| Increase request rate from 8 requests per second to 15 requests per second (ablation) | StarCoder-3B serving · StarCoder-3B | inflection rank: to around 12 | Helped |
| Use Fast LoRA instead of LoRA | MultiPL-E HumanEval · selected low-resource languages on StarCoder 15B and StarCoderBase 3B | Pass@1 | Mixed |
| Use Fast LoRA instead of IA3 | MultiPL-E HumanEval · selected low-resource programming languages | Pass@1 | Helped |
| Use Fast LoRA instead of LoRA | Common Voice speech recognition · Arabic, Czech, Lithuanian, Marathi, Mongolian, and Hindi | WER | Mixed |
| Use Fast LoRA instead of IA3 | Common Voice speech recognition · Arabic, Czech, Lithuanian, Marathi, Mongolian, and Hindi | WER | Helped |
| Apply adapters to every fully-connected layer rather than default PEFT modules (ablation) | Whisper fine-tuning · Common Voice | WER | Helped |
| Restrict IA3 to key and query projection layers instead of its default configuration (ablation) | IA3 · Multilingual code generation | Pass@1 | Hurt |
Surprising findings
- The torch.bmm LoRA implementation overtakes FLoRA in throughput once rank rises to about 8 on StarCoder 15B, so the advantage is limited to low ranks.
- The paper reports that the latency gain from FLoRA is theoretical at low request rates, since a 3B model at 8 requests/s does not saturate the server.
Limitations
- FLoRA still requires fine-tuning of adapters; deriving FLoRA weights from a trained LoRA is left to future work.
- Throughput advantage holds only at low ranks; BMM-based LoRA wins at higher ranks, and self-attention layers are not adapted.
- Experiments were run without FlashAttention and with vLLM 0.1.3 in float16, so results may shift with optimized kernels.
Experimental setup
Datasets: The Stack MultiPL-E Common Voice vLLM throughput benchmark dataset
Benchmarks: MultiPL-E Common Voice
Metrics: pass@1 word error rate throughput (tokens/s) latency (s/token)
Baselines: LoRA IA3 torch.bmm LoRA Whisper
Builds on: LoRA IA3 vLLM Orca
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.91)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.90)
- The Expressive Power of Low-Rank Adaptation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.