Research map / Large language models / Reasoning and chain of thought
BatchPrompt: Accomplish more with less
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BatchPrompt, which batches multiple data points per LLM prompt, and adds Batch Permutation and Ensembling (BPE) plus Self-reflection-guided Early Stopping (SEAS) to recover accuracy while cutting LLM calls and input tokens.
Problem
Single-data prompting wastes input tokens and LLM calls when the instructions and few-shot examples dominate the prompt, but naively batching multiple data points degrades accuracy, especially as batch size grows and with position-dependent answers.
Approach
Batch many data points per prompt, then recover quality via majority voting over repeated random permutations of the batch (self-ensembling), and use the LLM's own self-reported confidence to stop voting early on samples that are consistently answered as confident.
Key results
- At batch size 32 with BPE+SEAS, GPT-4 reaches 90.9% on BoolQ vs 90.6% SinglePrompt using 27.4% of tokens and 15.7% of LLM calls.
- On QQP at batch size 32, GPT-4 BPE+SEAS reaches 88.4% vs 87.2% SinglePrompt with 18.6% of tokens.
- On RTE at batch size 32, GPT-4 BPE+SEAS reaches 91.1% vs 91.5% SinglePrompt with 30.8% of tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply BatchPrompt + BPE with majority voting at batch size 32 and 9 voting rounds instead of SinglePrompt | GPT-4 · BoolQ | accuracy: 90.6% → 90.9% | Helped |
| Add confidence-guided early stopping (ablation) | BatchPrompt + BPE · BoolQ | accuracy: 90.6% → 90.9% | Mixed |
| Add confidence-guided early stopping instead of continuing voting for all examples (ablation) | BatchPrompt + BPE · BoolQ | tokens: 384946 → 107957 | Helped |
| Use negative few-shot examples with self-weighted voting (ablation) | BatchPrompt + BPE (sw-mv) · BoolQ | accuracy | Hurt |
| Increase batch size to 64 with one voting round (ablation) | BatchPrompt + BPE · BoolQ | accuracy: 90.6% → 72.8% | Hurt |
| Increase voting rounds from 1 to 5 (ablation) | BatchPrompt + BPE, batch size 32 · GSM8K | accuracy: 89.1% to 92.0% | Helped |
| Use batch size 32 and 5 voting rounds instead of SinglePrompt | BatchPrompt + BPE · GSM8K | accuracy: 94.9% → 92.0% | Hurt |
| Replace self-reflection-based early stopping with random dropping (ablation) | BatchPrompt + BPE · BoolQ, QQP, and RTE | accuracy | Helped |
| Increase voting rounds for batch size 32 (ablation) | BatchPrompt + BPE + SEAS · RTE | accuracy: 88.10 → 91.08 → 90.70 | Mixed |
Surprising findings
- Negative few-shot examples (sw-mv-neg) performed worse than standard self-weighted voting.
- Large batch sizes (64+) without BPE perform much worse than SinglePrompt, but recover with 5+ voting rounds.
Limitations
- Accuracy gains depend on high baseline accuracy; majority voting amplifies errors for low-accuracy tasks.
- Evaluated only on validation subsets (320 samples) with two GPT models; relies on closed APIs.
- Token savings vanish with many voting rounds unless SEAS is used; outputs for generation-style tasks vary too much for voting.
Experimental setup
Datasets: BoolQ QQP RTE GSM8K COPA MNLI
Benchmarks: BoolQ QQP RTE GSM8K COPA MNLI
Metrics: accuracy input token count number of LLM calls
Baselines: SinglePrompt naive BatchPrompt (no BPE) random drop
Builds on: self-consistency chain-of-thought reflexion
Closest papers by meaning
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.89)
- Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference (AAAI 2025 · similarity 0.89)
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.88)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.88)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.87)
- PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.