Research map / Architectures and efficiency / Pruning and efficient tuning
Dynamic Operator Optimization for Efficient Multi-Tenant LoRA Model Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dop, an automated search-based optimizer (hierarchical sketch/annotation search space plus evolutionary search guided by a cost model) that tunes the SGMV operator used in Punica for multi-tenant LoRA serving, reporting 1.30-1.46x throughput gains.
Problem
Multi-tenant LoRA serving with static, generic SGMV operator implementations gives suboptimal and hardware-/workload-dependent performance and low GPU utilization.
Approach
Treat SGMV implementation as an automated tensor-program search problem: build a hierarchical search space (high-level sketches from derivation rules, low-level random annotations) and refine it with evolutionary search guided by a learned throughput cost model, so the operator adapts to hardware and request distribution.
Key results
- Dop-optimized SGMV reduced operator latency by up to 56.21% on RTX 3090 (Distinct, batch 31) and 54.03% on A100 (Skewed, batch 45).
- Under the Skewed workload, Dop reached up to 1460 tokens/s on Llama-2-7B versus Punica's 1044 tokens/s.
- Dop improves throughput by 1.30-1.46x over the SOTA multi-tenant LoRA serving system (Punica).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize Punica's SGMV operator with Dop | Punica · Skewed, Llama-2 7B | throughput (tokens/second): 1044 tokens/second -> 1460 tokens/second | Helped |
| Optimize Punica's SGMV operator with Dop | Punica · Distinct, RTX 3090, batch size 31 | microbenchmark improvement over Punica: 56.21% | Helped |
| Optimize Punica's SGMV operator with Dop | Punica · Skewed, A100, batch size 45 | microbenchmark improvement over Punica: 54.03% | Helped |
| Add tile size mutation (ablation) | Dop-optimized SGMV · SGMV operator microbenchmark | throughput | Helped |
| Add parallel mutation (ablation) | Dop-optimized SGMV · SGMV operator microbenchmark | throughput | Helped |
| Add computation location mutation (ablation) | Dop-optimized SGMV · SGMV operator microbenchmark | throughput | Mixed |
Surprising findings
- Computation Location Mutation gave modest gains and in some cases slight performance degradation.
- In the Identical request pattern, gains over non-Punica baselines are smaller because the operation reduces to batched matrix multiplication.
Limitations
- Gains vary by request pattern and are smaller for the Identical pattern.
- Llama-2 13B/7B only; Llama-2 7B tested on A100 only due to RTX 3090 memory limits.
Experimental setup
Benchmarks: SGMV operator microbenchmark (Punica setup) text generation throughput on Llama-2 7B/13B
Metrics: latency throughput (tokens/second)
Baselines: Punica HuggingFace Transformers FasterTransformer vLLM DeepSpeed
Builds on: Punica Ansor TVM LoRA
Closest papers by meaning
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.87)
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized? (ICLR 2025 · similarity 0.87)
- LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently (ICML 2025 · similarity 0.87)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.87)
- LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization (ICLR 2025 · similarity 0.87)
- LoRA-GA: Low-Rank Adaptation with Gradient Approximation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.