Research map / Architectures and efficiency / Pruning and efficient tuning
Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BD-LoRA, which constrains two LoRA factors to be block-diagonal so tensor-parallel multi-LoRA serving needs no LoRA-specific communication, and shows up to 1.79x end-to-end speed-up over S-LoRA in vLLM at similar parameter counts and downstream quality.
Problem
Serving one base LLM with many LoRA adapters under tensor parallelism: S-LoRA's sharding of LoRA factors adds all-gather and all-reduce communication that becomes significant in practice even at small ranks.
Approach
Constrain LoRA factors B1 and A2 to be block-diagonal and shard them aligned with Megatron-LM TP, so LoRA computations need no extra communication; equivalent to independent rank r/N LoRA adapters per shard, trained from scratch with rsLoRA scaling alpha*sqrt(N)/sqrt(r).
Key results
- Up to 1.79x end-to-end speed-up (1.23x at matched rank-based setting) over S-LoRA with 0.87x adapter parameters for Llama-3.1-70B on eight A100 GPUs, with batch size 1, 1024 input and 128 output tokens.
- Up to 1.63x end-to-end speed-up over S-LoRA with 0.86x adapter parameters for Llama-3.1-8B on eight A100 GPUs.
- On OpenOrca and GLUE, BD-LoRA achieves performance comparable to standard LoRA at similar trainable parameter counts (e.g., Llama-3.2-1B GLUE average 74.8 for LoRA rank 512 vs. 74.8-75.0 for BD-LoRA rank 1024).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace S-LoRA sharding with block-diagonal LoRA sharding to eliminate LoRA communication | LoRA serving · Llama-3.1-70B on eight A100 GPUs | end-to-end speedup over S-LoRA: up to 1.79x | Helped |
| Replace S-LoRA sharding with block-diagonal LoRA sharding | Llama-3.1-8B serving · eight A100 GPUs | end-to-end speedup over S-LoRA: up to 1.63x | Helped |
| Use BD-LoRA instead of LoRA at a similar number of trainable parameters | Llama-3.2-1B · OpenOrca | perplexity: 3.09 -> 3.08 | No clear effect |
| Use BD-LoRA instead of LoRA at a similar number of trainable parameters | Llama-3.2-1B · GLUE average | average score: 74.8 -> 75.0 | No clear effect |
| Increase rank with rsLoRA scaling (ablation) | LoRA and BD-LoRA · OpenOrca and GLUE | downstream performance | Helped |
| Increase rank with standard scaling (ablation) | LoRA and BD-LoRA · OpenOrca and GLUE | downstream performance | No clear effect |
| Train BD-LoRA for TP=8 instead of TP=4 (ablation) | BD-LoRA · Llama-3.2-1B fine-tuning | downstream performance | No clear effect |
| Apply adapters only to attention instead of both attention and MLP (ablation) | LoRA and BD-LoRA · Llama-3.2-1B GLUE | average score | Mixed |
| Apply adapters only to MLP instead of both attention and MLP (ablation) | LoRA and BD-LoRA · Llama-3.2-1B GLUE | task score | Mixed |
| Use BD-LoRA instead of LoRA with attention-only or MLP-only adapters at lower ranks (ablation) | Llama-3.2-1B · GLUE | average score: less than 0.4% | Hurt |
| Increase TP degree (ablation) | Llama-3.1-8B serving · AWS p4d instance | run-time performance | Helped |
| Serve with BD-LoRA instead of S-LoRA when every request uses a different adapter (ablation) | LoRA serving · Llama-3.1-70B | speedup: up to 1.78x | Helped |
Surprising findings
- BD-LoRA sometimes slightly outperforms standard LoRA on OpenOrca and average GLUE, though confidence intervals largely overlap.
- Under standard alpha/r scaling, LoRA performance does not improve with rank, whereas with rsLoRA scaling it does.
Limitations
- The TP degree N must be fixed at training time, since block-diagonal shape depends on N.
- Existing standard LoRA adapters cannot be reused; BD-LoRA adapters must be trained from scratch.
- Speed-ups over NFS-LoRA shrink at small ranks, and the paper's main run-time experiments use a single cached adapter, an artificial setting.
Experimental setup
Datasets: OpenOrca GLUE
Benchmarks: OpenOrca GLUE
Metrics: perplexity accuracy Matthews correlation Pearson correlation end-to-end latency throughput decoding latency prefill latency
Baselines: S-LoRA NFS-LoRA standard LoRA
Builds on: LoRA S-LoRA rsLoRA Megatron-LM vLLM
Closest papers by meaning
- From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control (ICML 2025 · similarity 0.86)
- Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead (ICML 2025 · similarity 0.84)
- LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization (ICLR 2025 · similarity 0.84)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.84)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.84)
- Dynamic Operator Optimization for Efficient Multi-Tenant LoRA Model Serving (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.