Research map / Large language models / Reasoning and chain of thought
Fast Inference for Augmented Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAMPS, a scheduler for API-augmented LLM requests that ranks requests by predicted memory-over-time (including API handling strategy: preserve/discard/swap) and adds starvation prevention, implemented on vLLM.
Problem
Size-based and FCFS scheduling perform poorly for API-augmented LLM requests because output length does not reflect total time, and KV-cache memory held during API calls causes head-of-line blocking.
Approach
Predict pre-API output length and API duration, pick the memory handling strategy (preserve, discard-and-recompute, swap) minimizing memory waste before scheduling, then rank requests by their predicted integral of memory over time rather than by length.
Key results
- Reduces end-to-end latency by 27%-85% and TTFT by 4%-96% compared to INFERCEPT on Vicuna 13B and GPT-J 6B across datasets, with greater gains over vLLM.
- Response-length predictor on ToolBench reaches 68.5% Acc-5, 78.3% Acc-15 and MAE 3.06 words, with 13.7 ms per-input prediction on an A100.
- On ToolBench at rate 3 with GPT-J 6B, reduces mean TTFT by 87.04% vs INFERCEPT and latency by 27.24%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Integrate request length and API handling strategy into scheduling rather than using FCFS | LAMPS on augmented-LLM requests · Multi-API, illustrative example | average request completion time: 11.66 units -> 10 units | Helped |
| Use SJF by total length instead of the memory-aware optimized policy | Augmented-LLM scheduler · Multi-API, illustrative example | average request completion time: 10 units -> 11 units | Hurt |
| Add predicted API handling while retaining FCFS scheduling (ablation) | vLLM · Multi-API, Vicuna 13B | performance across reported metrics | No clear effect |
| Add the LAMPS scheduling policy to predicted API handling (ablation) | LAMPS w/o scheduling · Multi-API, Vicuna 13B | performance across reported metrics | Helped |
| Introduce a starvation-prevention threshold of 100 (ablation) | LAMPS scheduler · Multi-API, GPT-J 6B | tail latency and throughput: 100 | Helped |
| Increase prediction error (ablation) | LAMPS predictions · Multi-API, GPT-J 6B | median latency: 5%, 10%, 30%, 50% | Hurt |
| Increase prediction error (ablation) | LAMPS predictions · Multi-API, GPT-J 6B | throughput: 5%, 10%, 30%, 50% | Hurt |
| Optimize end-to-end latency rather than prioritizing throughput | LAMPS · ToolBench | throughput vs. INFERCEPT | Hurt |
Surprising findings
- Naive SJF ordering by total length (output plus API time) performed worse than plain SJF in the motivating example.
- LAMPS shows higher P99 latency and TTFT growth at high request rates due to starvation effects, though absolute values remain lower than baselines.
- Throughput gains over INFERCEPT are smaller on ToolBench, where latency gains come with slight throughput degradation.
Limitations
- Assumes a single API per segment; multi-API cumulative memory consumption is not modeled.
- Prediction errors degrade performance when large; output length predictor accuracy is moderate (Acc-5 68.5%).
- Higher tail latency and TTFT at high request rates due to starvation of long/memory-heavy requests.
Experimental setup
Datasets: INFERCEPT dataset ToolBench
Benchmarks: INFERCEPT dataset (single-API, multi-API) ToolBench
Metrics: end-to-end latency time-to-first-token throughput P99 latency Acc-5 Acc-15 MAE
Baselines: vLLM INFERCEPT
Builds on: INFERCEPT vLLM Orca
Closest papers by meaning
- AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving (ICML 2026 · similarity 0.94)
- Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · similarity 0.90)
- LLM Query Scheduling with Prefix Reuse and Latency Constraints (NeurIPS 2025 · similarity 0.89)
- Efficient Multi-round LLM Inference over Disaggregated Serving (ICML 2026 · similarity 0.89)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.89)
- Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.