Research map / Large language models / Instruction and fine-tuning
Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Loquetier is a virtualized multi-LoRA framework that unifies LoRA fine-tuning and serving on a shared Llama3-8B base via a Virtualized Module and an SMLM kernel that merges training and inference paths in one forward pass.
Problem
Existing LoRA systems either serve adapters or fine-tune them, but not both efficiently in one runtime; they fuse adapters into monolithic instances, degrade decoding during concurrent fine-tuning, and require halting for task switching.
Approach
Isolate PEFT modifications with runtime-created virtual modules over a shared base model, and adapt the Punica segmented multi-LoRA kernel to per-linear-layer processing so fine-tuning, evaluation, prefilling and decoding requests share one forward pass, with per-adapter losses summed for a shared backward pass.
Key results
- Up to 3.0x the throughput of FlexLLM on inference-only tasks at high request arrival rates.
- 46.4x higher SLO attainment than PEFT on unified fine-tuning and inference tasks.
- Overall SLO attainment of 92.37% on a simulated 120-minute real-world BurstGPT workload.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Loquetier instead of FlexLLM for inference-only serving | Llama3-8B · Inference-only tasks | throughput: up to 3.0× | Helped |
| use Loquetier instead of PEFT for inference-only serving | Llama3-8B · Inference-only tasks | SLO attainment: 20.8× higher | Helped |
| use Loquetier instead of PEFT for unified fine-tuning and inference | Llama3-8B · Unified fine-tuning and inference | SLO attainment: 46.4× higher | Helped |
| run fine-tuning and inference in Loquetier's unified runtime | Llama3-8B · Unified fine-tuning and inference | fine-tuning efficiency: about 40% | Mixed |
| use Loquetier under simulated real-world workload | Llama3-8B · BurstGPT simulated real-world workload | SLO attainment: 92.37% | Helped |
| use Loquetier instead of PEFT for loading | Llama3-8B · Model loading | total loading time: 4.791 s -> 5.336 s | Hurt |
| use Loquetier instead of FlexLLM for loading | Llama3-8B · Model loading | total additional storage: 15.00 GB -> 0 B | Helped |
Surprising findings
- S-LoRA produced frequent NaN/Inf outputs and often emitted EOS directly, so its SLO numbers were judged overstated.
- FlexLLM's backward pass failed on unimplemented gradient ops (GELU, RELU, etc.); the authors patched it themselves.
Limitations
- Backward pass does not use the SMLM kernel; fine-tuning falls back to standard PyTorch autograd path and is slightly slower than PEFT.
- Only LoRA-like PEFT methods evaluated; support for other PEFT variants is future work.
- Experiments use a single base model (Llama3-8B) and no error bars are reported.
Experimental setup
Datasets: ShareGPT Alpaca GSM8K BurstGPT
Benchmarks: ShareGPT Alpaca GSM8K BurstGPT
Metrics: SLO attainment decode throughput (DTPS) finetune throughput (FTPS) evaluate throughput (ETPS) request throughput (RPS) model loading time
Baselines: FlexLLM S-LoRA+PEFT PEFT
Builds on: Punica FlashInfer S-LoRA PEFT Hugging Face Transformers Trainer
Closest papers by meaning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.94)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.93)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.92)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.92)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.