Research map / Architectures and efficiency / RNNs and normalization
Auto-reconfiguration for Latency Minimization in CPU-based DNN Serving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Packrat is a TorchServe extension that, for CPU DNN serving, runs multiple thin model instances with fewer threads and smaller batches, choosing the configuration via profiling plus a 2D knapsack-style dynamic program and reconfiguring online without downtime.
Problem
CPU-based DNN serving latency: intra-operator parallelism over many threads gives diminishing returns, and the best mix of instances, threads, and batch size is workload- and hardware-dependent and hard to choose manually.
Approach
Instead of one fat instance using all threads, run several thin instances each with fewer threads and a partitioned batch; an optimizer built on a small set of single-instance profiles picks the <instances, threads, batch> configuration via dynamic programming, and active-passive scaling reconfigures online.
Key results
- Packrat improves average end-to-end inference latency over fat-instance TorchServe by 1.43x to 1.83x averaged across batch sizes, with max speedup 1.72x to 2.09x.
- Microbenchmark mean speedups across batch sizes are 1.53x for ResNet-50, 1.52x for Inception-v3, 1.18x for GPT-2 and 1.13x for BERT.
- Profiling with powers-of-two batch sizes reduces profiled configurations from 16,384 to 176 for n=10, T=16, cutting profiling time from about 30 days to a few hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the fat-instance configuration with Packrat-selected multi-instance execution | PyTorch inference · ResNet-50, averaged across batch sizes | speedup over fat-instance baseline: 1.53× | Helped |
| Replace the fat-instance configuration with Packrat-selected multi-instance execution | PyTorch inference · Inception-v3, averaged across batch sizes | speedup over fat-instance baseline: 1.52× | Helped |
| Replace the fat-instance configuration with Packrat-selected multi-instance execution | PyTorch inference · GPT-2, averaged across batch sizes | speedup over fat-instance baseline: 1.18× | Helped |
| Replace the fat-instance configuration with Packrat-selected multi-instance execution | PyTorch inference · BERT, averaged across batch sizes | speedup over fat-instance baseline: 1.13× | Helped |
| Automatically reconfigure instances and threads using Packrat | TorchServe DNN serving · Across models and batch sizes | inference latency speedup: 1.43 to 1.83× | Helped |
| Run multiple thin instances concurrently rather than profiling each instance in isolation | Packrat multi-instance execution · ResNet-50, T=16, B=256 | latency: 1224 ms to 1600 ms | Hurt |
Surprising findings
- Actual speedup is consistently below the optimizer's expected speedup because concurrent instances suffer license-based CPU downclocking and memory latency contention, though the gap is constant across models so selected configurations remain unchanged.
- Single-threaded instances (one per core) often perform worse than the fat-instance baseline due to cross-instance interference.
Limitations
- Optimizer uses profiles measured in isolation and ignores multi-instance contention, causing a gap between expected and actual speedup.
- Reconfiguration takes several seconds and transient latency rises during the switch, so workloads must be stable.
- Evaluated on a single two-socket Xeon server (plus two additional machines in the appendix) with only four models.
Experimental setup
Benchmarks: ResNet-50 microbenchmark Inception-v3 microbenchmark GPT-2 microbenchmark BERT microbenchmark end-to-end TorchServe latency and throughput
Metrics: latency speedup throughput speedup
Baselines: fat-instance execution (TorchServe default) single-threaded instances (one per core) PyTorch eager mode
Builds on: TorchServe Intel Extension for PyTorch multi-dimensional knapsack
Closest papers by meaning
- Practical Performance Guarantees for Pipelined DNN Inference (ICML 2024 · similarity 0.86)
- The streaming rollout of deep networks - towards fully model-parallel execution (NeurIPS 2018 · similarity 0.86)
- Efficient Distributed Inference of Deep Neural Networks via Restructuring and Pruning (AAAI 2023 · similarity 0.86)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.85)
- Don’t Think It Twice: Exploit Shift Invariance for Efficient Online Streaming Inference of CNNs (AAAI 2025 · similarity 0.84)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.