Research map / Large language models / Reasoning and chain of thought
Think before you speak: Training Language Models With Pause Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pause-training: appending learnable <pause> tokens during pretraining, finetuning, and inference so the model gets extra computation before emitting its answer, showing gains on several downstream tasks for a 1B model when pauses are used in both pretraining and finetuning.
Problem
Transformer language models have a per-layer computation budget tied to the number of preceding tokens, so the next-token computation is constrained; it is unclear whether delaying output via extra input tokens helps and how to train for it.
Approach
Append a single learnable pause token (repeated M times) to the input and ignore outputs until the last pause; inject pauses during pretraining (randomly inserted, loss skipped on pause targets), finetuning, and inference, so the model learns to use the delayed computation.
Key results
- For the 1B model, pause-pretraining plus pause-finetuning improves SQuAD EM from 36.4 to 55.9 (about 18 points).
- CommonSenseQA EM rises from 26.9 to 34.8 and GSM8k accuracy from 7.5 to 8.5 for the 1B model with pauses in both stages.
- Pause-finetuning only on a standard pretrained model gives mixed, mostly mild gains and even hurts GSM8k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pause tokens during both pretraining and downstream finetuning, with pauses at inference | 1B decoder-only model · SQuAD | EM: 36.4 ± 2.5 -> 55.9 ± 1.0 | Helped |
| Use pause tokens during both pretraining and downstream finetuning, with pauses at inference | 1B decoder-only model · GSM8k | accuracy: 7.5 ± 0.5 -> 8.5 ± 0.9 | Helped |
| Use pause tokens during both pretraining and downstream finetuning, with pauses at inference | 1B decoder-only model · CommonSenseQA | EM: 26.9 ± 2.9 -> 34.8 ± 1.2 | Helped |
| Use pause tokens during both pretraining and downstream finetuning, with pauses at inference | 1B decoder-only model · HellaSwag | F1: 37.8 ± 0.1 -> 37.7 ± 0.2 | Hurt |
| Add pause tokens during finetuning only, without pause-pretraining (ablation) | 1B decoder-only model · GSM8k | accuracy: 7.5 ± 0.5 -> 6.9 ± 1.0 | Hurt |
| Add pause tokens during finetuning only, without pause-pretraining (ablation) | 1B decoder-only model · SQuAD | EM: 36.4 ± 2.5 -> 40.2 ± 3.2 | Helped |
| Pause-pretrain but use standard finetuning and inference (ablation) | 1B decoder-only model · GSM8k | accuracy: 7.5 ± 0.5 -> 7.7 ± 0.5 | No clear effect |
| Pause-pretrain but use standard finetuning and inference (ablation) | 1B decoder-only model · SQuAD | EM: 36.4 ± 2.5 -> 38.4 ± 2.9 | No clear effect |
| Increase pause count from 10 to 50 during finetuning and inference (ablation) | Pause-pretraining + pause-finetuning · GSM8k | accuracy: 8.5 ± 0.9 -> 7.7 ± 0.3 | Hurt |
| Use 50 rather than 10 pauses during finetuning and inference (ablation) | Pause-pretraining + pause-finetuning · SQuAD | EM: 51.7 ± 2.3 -> 55.9 ± 1.0 | Helped |
| Prepend pauses rather than append them (ablation) | Pause-pretraining + pause-finetuning · GSM8k | accuracy: 8.5 ± 0.9 -> 8.0 ± 0.4 | Hurt |
| Prepend pauses rather than append them (ablation) | Pause-pretraining + pause-finetuning · SQuAD | EM: 55.9 ± 1.0 -> 44.0 ± 3.2 | Hurt |
Surprising findings
- Filler periods at inference on a standard model give no gains, while pause-pretrained models do improve.
- Providing zero pauses at inference collapses performance of pause-pretrained and pause-finetuned models, despite graceful degradation with fewer nonzero pauses.
Limitations
- Pause-pretraining is expensive and not practical for most users.
- Only tested on 1B and 130M decoder-only models with C4 pretraining; no encoder-decoder or other scales.
- Zero-pause inference breaks pause-trained models; the number of pauses must be chosen per task.
Experimental setup
Datasets: C4 GSM8k SQuAD CoQA CommonSenseQA PhysicalIQA LAMBADA HellaSwag WebQuestions Natural Questions
Benchmarks: GSM8k SQuAD CoQA CommonSenseQA PhysicalIQA LAMBADA HellaSwag WebQuestions Natural Questions
Metrics: exact match accuracy F1 Rouge1
Baselines: standard pretraining and standard finetuning filler-period inference (Pfau et al.) standard pretraining with pause finetuning
Builds on: chain-of-thought prompting memory tokens Universal Transformer
Closest papers by meaning
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.89)
- Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.88)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.87)
- On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study (NeurIPS 2025 · similarity 0.87)
- BatchPrompt: Accomplish more with less (ICLR 2024 · similarity 0.87)
- Not All Tokens Are What You Need for Pretraining (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.