Research map / Large language models / Language, safety and interpretability
Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skeleton-of-Thought, a prompting method that has an LLM first write an answer skeleton and then expands each point in parallel (API calls or batched decoding) to cut generation latency, with a router variant that falls back to normal decoding for unsuitable questions.
Problem
LLM inference latency is high because almost all state-of-the-art LLMs decode answers sequentially token by token.
Approach
Prompt the LLM itself to plan a short answer skeleton, then expand every skeleton point in parallel and concatenate the results, reducing end-to-end latency without changing model, system, or hardware; a router (GPT-4 prompting or a trained RoBERTa) triggers SoT only for suitable questions.
Key results
- SoT gives over 2x speed-up (up to 2.39x) on 8 of 12 evaluated models on Vicuna-80.
- Under GPT-4 judging with FastChat, SoT is not worse than normal decoding in around 60% of cases, with win rates close to lose rates.
- SoT-R with a trained RoBERTa router substantially improves answer quality on questions unsuitable for SoT (e.g., coding, math, writing) while keeping speed-ups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate an answer skeleton, then expand its points in parallel instead of decoding sequentially | LLMs evaluated with Skeleton-of-Thought · Vicuna-80 | speedup: up to 2.39× | Helped |
| Generate an answer skeleton, then expand its points in parallel | Claude · Question in the latency example | latency: 22 seconds -> 12 seconds | Helped |
| Generate an answer skeleton, then expand its points in parallel | Vicuna-33B V1.3 · Question in the latency example | latency: 43 seconds -> 16 seconds | Helped |
| Use Skeleton-of-Thought rather than normal sequential generation | LLMs evaluated on Vicuna-80 · Vicuna-80, general answer quality | answer quality | Mixed |
| Use Skeleton-of-Thought rather than normal sequential generation | LLMs evaluated with LLMZoo · Assistant-style questions | diversity | Helped |
| Use Skeleton-of-Thought rather than normal sequential generation | LLMs evaluated with LLMZoo · Assistant-style questions | relevance | Helped |
| Use Skeleton-of-Thought rather than normal sequential generation | LLMs evaluated with LLMZoo · Assistant-style questions | immersion | Hurt |
| Use Skeleton-of-Thought rather than normal sequential generation | LLMs evaluated with LLMZoo · Assistant-style questions | coherence | Hurt |
| Use Skeleton-of-Thought on questions suited to independent point-wise expansion | LLMs evaluated on Vicuna-80 · Knowledge, generic, common-sense, roleplay, and counterfactual | speedup: 1.89× to 2.33× | Helped |
| Use Skeleton-of-Thought on questions requiring step-by-step reasoning | LLMs evaluated on Vicuna-80 · Math, coding, writing, and Fermi questions | answer quality | Hurt |
| Add a router that selects SoT for suitable questions and falls back to normal generation otherwise | Skeleton-of-Thought with router · Vicuna-80 | answer quality | Helped |
| Add a router that selects SoT for suitable questions and falls back to normal generation otherwise | Skeleton-of-Thought with router · Vicuna-80 | speedup | Hurt |
Surprising findings
- Some SoT answers are rated better than normal generation on diversity and relevance, while coherence and immersion are hurt.
- Prompting and trained routers can even surpass a human router on some categories such as roleplay.
Limitations
- Answer quality evaluation relies on GPT-4 judges with a limited prompt set and no human evaluation.
- SoT ignores dependencies between points and is unsuitable for step-by-step reasoning questions such as math and coding.
- For API models, SoT increases prefilling token counts and therefore cost.
Experimental setup
Datasets: Vicuna-80 WizardLM LIMA
Benchmarks: Vicuna-80 WizardLM
Metrics: speed-up net win rate FastChat general quality score LLMZoo detailed metrics
Baselines: normal sequential decoding GPT-4 prompting router human router
Builds on: chain-of-thought speculative decoding FastChat evaluation LLMZoo evaluation
Closest papers by meaning
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention (NeurIPS 2025 · similarity 0.89)
- PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries (NeurIPS 2025 · similarity 0.88)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.87)
- Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning (ICLR 2024 · similarity 0.87)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.