Research map / Large language models / Instruction and fine-tuning
Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRA-Inlaid, a multi-LoRA serving system with MLGPTQ, a joint quantization method that yields one shared quantized base model for multiple LoRA adapters, plus dynamic task addition and output-length-guided grouped scheduling.
Problem
Quantized base models (GPTQ/AWQ) are task-specific when calibrated with LoRA adapters, so they cannot be shared across tasks in multi-LoRA serving; existing systems also lack dynamic task addition and ignore workload differences across tasks during scheduling.
Approach
MLGPTQ quantizes the base model jointly for all tasks by loading each task's LoRA during the forward pass and max-aggregating per-task inverse Hessian diagonals to choose weight updates; incremental quantization caches the aggregated Hessian so new tasks can be added without interrupting service; scheduling uses output-length prediction with SRTF and task grouping to limit the number of LoRA adapt
Key results
- LoRA-Inlaid outperforms S-LoRA and vLLM by up to 1.58x throughput, 1.76x average latency, 2x JCT, and 10x SLO attainment.
- At 4-bit, MLGPTQ's average accuracy drop is 1.70%, versus 4.72% for GPTQ-tweaked, 4.50% for AWQ-tweaked, and 4.02% for RTN.
- Dynamic task addition reduces throughput by 10-13% while new tasks are added, and layer-wise incremental quantization uses 2.5GB peak memory versus 9.2GB for full quantization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MLGPTQ instead of GPTQ with mixed calibration data for shareable 4-bit quantization | LLaMA2 multi-task quantization · GSM8k | Accuracy (%): 31.51 (0) -> 31.62 (0) | Helped |
| Use 4-bit MLGPTQ quantization rather than no quantization | LLaMA2 multi-task quantization · GSM8k | Accuracy (%): 32.14 (0) -> 31.62 (0) | Hurt |
| Use MLGPTQ rather than S-LoRA | LLaMA2-7B serving · RTX 4090, 2 tasks, request rate 5 | Throughput (reqs/s): 2.93 -> 3.89 | Helped |
| Use MLGPTQ rather than S-LoRA | LLaMA2-7B serving · RTX 4090, 1000 tasks, request rate 5 | Throughput (reqs/s): 2.78 -> 3.42 | Helped |
| Add task grouping to multi-task scheduling (ablation) | LoRA-Inlaid · 100 tasks | SLO Attainment: +1.16× on average | Helped |
| Add output length prediction to multi-task scheduling (ablation) | LoRA-Inlaid · 100 tasks | SLO Attainment: +1.23× on average | Helped |
| Add prediction-based SRTF scheduling (ablation) | LoRA-Inlaid · 100 tasks | SLO Attainment: +2.27× on average | Helped |
| Add multi-task quantization (ablation) | LoRA-Inlaid serving LLaMA2-7B · Serving workloads | SLO Attainment: 39% | Helped |
| Add multi-task quantization (ablation) | LoRA-Inlaid serving LLaMA2-13B · Serving workloads | memory feasibility | Helped |
| Include the target task's calibration set and LoRA adapter during quantization (ablation) | MLGPTQ · Machine translation tasks | S_BLEU | Helped |
| Add incremental quantization while serving continues | LoRA-Inlaid · LLaMA2-7B, online task addition | Throughput: 10%-13% of degradation | Hurt |
Surprising findings
- Without quantization, serving LLaMA2-13B runs out of memory, so multi-task quantization is required to serve the larger model under the 24GB limit.
Limitations
- Quantization does not detect malicious or poisoning tasks.
- Scheduling ignores fairness among tasks.
- Supports only language tasks, not multimodal ones.
Experimental setup
Datasets: OPUS (six translation pairs) XLSum QTSumm tiny-codes GSM8K medical_meadow_mmmlu malicious-600k
Benchmarks: OPUS XLSum QTSumm tiny-codes GSM8K medical_meadow_mmmlu malicious-600k
Metrics: SacreBLEU ROUGE-1 accuracy throughput average latency job completion time SLO attainment
Baselines: RTN GPTQ (tweaked) AWQ (tweaked) vLLM S-LoRA Skip-join MLFQ
Builds on: GPTQ LoRA S-LoRA Punica FastServe
Closest papers by meaning
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.90)
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.90)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.89)
- LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · similarity 0.89)
- MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models (ICLR 2025 · similarity 0.89)
- BSLoRA: Enhancing the Parameter Efficiency of LoRA with Intra-Layer and Inter-Layer Sharing (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.