Research map / Large language models / Reasoning and chain of thought
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Perception Only prompting with instruction-tuned length predictor (ablation) | length predictor on Vicuna-7B · Alpaca 10k | length error: 193 -> 63 | Helped |
| use instruction tuning vs pooling+MLP/[LEN]-token for length prediction (ablation) | sequence scheduling · Alpaca 10k | throughput (samples/s): 1.96/2.10 -> 2.27 | Helped |
| predict max length instead of mean length (ablation) | instruction-tuned predictor · Alpaca 10k | throughput improvement: +45 -> +86 | Helped |
| add binning of target length (with FCR+VBS) (ablation) | sequence scheduling · Alpaca 10k | throughput (samples/s): 2.22 -> 2.27 | Helped |
| add binning alone (ablation) | sequence scheduling · Alpaca 10k | throughput (samples/s): 1.73 -> 1.58 | Hurt |
| add Failure Collection and Recomputation (with binning) (ablation) | sequence scheduling · Alpaca 10k | throughput (samples/s): 1.58 -> 1.86 | Helped |
| add Variable Batch Size (with binning+FCR) (ablation) | sequence scheduling · Alpaca 10k | throughput (samples/s): 1.86 -> 2.27 | Helped |
| add sequence scheduling with instruction-tuned predictor | Vicuna-7B inference · Instruction-in-Wild | throughput (samples/s): 0.78 -> 1.24 | Helped |
| use Perception in Advance prompting on small models | Vicuna-7B · response generation | response quality | Hurt |
| increase group size | sequence scheduling · Alpaca 10k | throughput | Helped |
Closest papers by meaning
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.93)
- Efficient LLM Scheduling by Learning to Rank (NeurIPS 2024 · similarity 0.92)
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.92)
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.91)
- Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.91)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.