Research map / Large language models / Reasoning and chain of thought
Prompt Sketching for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces prompt sketching, which frames template-guided LLM inference as segmented sequence decoding and proposes two sketch-aware beam search decoders (Var and BeamVar) that jointly optimize template likelihood.
Problem
Sequential multi-call prompting (e.g., chain-of-thought, stop-and-go templating) leaves the model unaware of follow-up template parts, producing disconnected and wordy intermediate outputs and suboptimal reasoning.
Approach
Treat an entire prompt template with multiple variables as one segmented sequence and adapt decoding (variable-level and variable-grid beam search) to score follow-up deterministic instructions during generation, so the overall template likelihood is optimized end-to-end.
Key results
- With text-davinci-003 (zero-shot), prompt sketching with BeamVar decoding reaches 0.77 on StrategyQA vs 0.72 for CoT beam search and 0.47 on AQuA vs 0.37 for CoT, improving accuracy by up to ~10 points.
- On Llama-2 Chat 13B, BeamVar sketching improves GSM8K from 0.276 (CoT ArgMax) to 0.353 and Date Understanding from 0.591 to 0.659.
- On a sudoku task, ArgMax sketching solves 1/10 puzzles while BeamVar and Var solve 6/10 and 7/10 respectively; dungeon escape with BeamVar succeeds 10/10 with about 2.4 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use prompt sketching with BeamVar instead of sequential Chain-of-Thought beam decoding | text-davinci-003 · AQuA | accuracy: 0.37 -> 0.47 | Helped |
| Use prompt sketching with BeamVar instead of sequential Chain-of-Thought beam decoding | text-davinci-003 · StrategyQA | accuracy: 0.72 -> 0.77 | Helped |
| Use prompt sketching with Var instead of sequential Chain-of-Thought decoding | text-davinci-003 · Matrix Shapes | accuracy: 0.77 -> 0.85 | Helped |
| Use prompt sketching with BeamVar instead of sequential Chain-of-Thought decoding | Llama-2 Chat 13B · AQuA | accuracy: 0.2908 -> 0.255 | Hurt |
| Use prompt sketching with BeamVar instead of sequential Chain-of-Thought decoding | Llama-2 Chat 13B · GSM8K | accuracy: 0.276 -> 0.353 | Helped |
| Use sketch-aware BeamVar instead of sketch ArgMax (ablation) | text-davinci-003 prompt sketching · Information Essentiality | accuracy: 0.01 -> 0.25 | Helped |
| Use sketch-aware BeamVar instead of sketch ArgMax (ablation) | text-davinci-003 prompt sketching · AQuA | accuracy: 0.40 -> 0.47 | Helped |
| Use sketch-aware Var instead of sketch ArgMax (ablation) | text-davinci-003 prompt sketching · Tracking Shuffled Objects | accuracy: 0.64 -> 0.66 | Helped |
| Use sketch-aware Var instead of sketch ArgMax (ablation) | text-davinci-003 prompt sketching · GSM8K | accuracy: 0.57 -> 0.59 | Helped |
| Use two-shot demonstrations instead of zero-shot sketching (ablation) | text-davinci-003 prompt sketching · Tracking Shuffled Objects | accuracy: 0.66 -> 0.57 | Hurt |
| Use two-shot demonstrations instead of zero-shot sketching (ablation) | text-davinci-003 prompt sketching · AQuA | accuracy: 0.47 -> 0.46 | Mixed |
| Use prompt sketching with Var instead of sketch ArgMax (ablation) | text-davinci-003 · Sudoku (3x3) | puzzles solved: 1/10 -> 7/10 | Helped |
Surprising findings
- Sequential ArgMax sketching solves only 1 of 10 sudoku puzzles with text-davinci-003 when the template is not reordered, while reordering yields perfect accuracy with ArgMax.
- Two-shot demonstrations decreased performance on interleaved reasoning tasks such as Tracking Shuffled Objects relative to zero-shot sketching.
Limitations
- Results with text-davinci-003 use only 100 samples per configuration, limiting statistical power.
- Smaller models (text-curie-001) show near-random reasoning, so sketching benefits depend on model scale.
- Sketch design remains sensitive to wording and requires iteration; overly restrictive sketches can hurt performance.
Experimental setup
Datasets: Date Understanding (BIG-bench) Information Essentiality AQuA StrategyQA Multistep Arithmetic (BIG-bench) GSM8K Tracking Shuffled Objects (BIG-bench) Matrix Shapes (BIG-bench)
Benchmarks: Date Understanding Information Essentiality AQuA StrategyQA Multistep Arithmetic GSM8K Tracking Shuffled Objects Matrix Shapes Sudoku (3x3) Dungeon Escape
Metrics: accuracy success rate steps to exit Clopper-Pearson 95% confidence intervals
Baselines: answer-only prompting chain-of-thought prompting stop-and-go inference ArgMax decoding beam search
Builds on: chain-of-thought prompting zero-shot CoT LMQL grid beam search dynamic beam allocation
Closest papers by meaning
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.90)
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.89)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.89)
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.89)
- Recursive Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.