Research map / Large language models / Instruction and fine-tuning
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes joint compression of many LoRA adapters into a shared basis with per-LoRA scaling matrices (JD-Full/JD-Diag), plus clustering, and integrates it into vLLM to serve over 1000 LoRAs with high throughput.
Problem
Serving thousands of LoRA adapters requires constant loading and offloading of adapters from GPU memory, which degrades throughput when the adapter count is large.
Approach
Jointly compress a collection of LoRA products B_i A_i into a shared orthogonal basis U, V with small LoRA-specific core matrices Sigma_i (solved by alternating joint diagonalization), and cluster LoRAs so each cluster gets its own basis, keeping all compressed adapters resident on the GPU.
Key results
- JD-based compression with clustering keeps Rouge-L performance at 99%+ of uncompressed LoRAs for settings with reconstruction loss below 0.6, sometimes outperforming originals.
- When serving over 1000 LoRAs in vLLM, compression increases throughput 1.6x and maintains 80% of the throughput of serving a single LoRA or base model.
- Reconstruction error correlates with a decreasing, roughly exponential trend with Rouge-L performance, and moderate reconstruction error (~60%) does not hurt performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly compress LoRAs before serving | vLLM multi-LoRA serving · Over 1000 LoRAs | throughput: 1.6\!\times | Helped |
| Jointly compress LoRAs before serving | Mistral-7B-Instruct serving · Over 1000 LoRAs | throughput relative to serving a single LoRA: 80\% | Helped |
| Fine-tune with LoRA rather than use the base model | Mistral-7B-Instruct-v0.2 · 1000 natural instruction tasks | Rouge-L: 20.62 \pm 18.21 -> 67.80 \pm 30.15 | Helped |
| Use JD-Full rather than JD-Diag (ablation) | Joint diagonalization compression · LoRA collections | task performance | Helped |
| Use JD-Full rather than JD-Diag (ablation) | Joint diagonalization compression · Fewer than 100 LoRAs | task performance | No clear effect |
| Add clustering to JD-Full compression (ablation) | JD-Full · 500–1000 LoRAs | task performance | Helped |
| Add clustering to joint diagonalization (ablation) | Joint diagonalization compression · LoRA collections at the same reconstruction error | task performance | Helped |
| Compress each LoRA independently with r-SVD rather than jointly (ablation) | LoRA collections · Natural instruction tasks | task performance | Hurt |
| Jointly compress adapters A and B (ablation) | Compressed adapter A · Task T_B | performance on T_B | No clear effect |
Surprising findings
- Moderate reconstruction error (around 60%) does not damage performance and may slightly outperform zero-error compression.
- Compressed LoRAs often outperform the original uncompressed LoRAs on generalization metrics.
Limitations
- Joint compression alone is insufficient for hundreds of thousands of LoRAs; scheduling would be needed.
- Reconstruction error is a proxy for downstream performance and the relationship is non-linear.
- Experiments use a single base model (Mistral-7B-Instruct-v0.2) and in-distribution tasks.
Experimental setup
Datasets: Natural Instructions (1000 tasks)
Benchmarks: Natural Instructions (10 main tasks)
Metrics: Rouge-L Rouge-1 exact match cross-entropy loss agreement throughput (requests per second)
Baselines: S-LoRA vLLM multi-LoRA r-SVD uncompressed LoRA
Builds on: LoRA S-LoRA Punica vLLM VeRA
Closest papers by meaning
- NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.90)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering (ICLR 2025 · similarity 0.89)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.88)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.87)
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.