Research map / Large language models / Language, safety and interpretability
Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Scaffold-BPE, a parameter-free modification of BPE that marks low-frequency intermediate "scaffold" tokens during training and demolishes them at encoding time, improving LLM pretraining and machine translation over original BPE.
Problem
Original BPE keeps intermediate tokens (scaffold tokens) that rarely occur on their own but are components of longer frequent tokens, creating token frequency imbalance that hinders learning of those tokens and wastes vocabulary.
Approach
During BPE training, tokens whose frequency drops below the frequency of the next merge candidate are marked as scaffold tokens and excluded from the final vocabulary; at encoding time they are used as temporary merge intermediates (Scaffolding) and then decomposed into their shortest non-scaffold subsequences (Demolishing).
Key results
- Scaffold-BPE improves average 0/5-shot commonsense accuracy over original BPE at all scales, e.g., 6.7B model gains 2.08pp on BoolQ and 2.90pp on OpenBookQA.
- On closed-book QA with the 6.7B models, TriviaQA exact match rises from 15.63 to 18.86 and WebQuestions from 8.56 to 9.89.
- On WMT'14 En-De and En-Fr, Scaffold-BPE reaches BLEU 29.76 and 43.81 versus 29.31 and 43.20 for original BPE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Original BPE with Scaffold-BPE | 468M language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 58.64 -> 60.52; 40.78 -> 41.68; 30.50 -> 32.20; 66.57 -> 68.69; 43.40 -> 44.09; 62.77 -> 63.04; 53.00 -> 54.22 | Helped |
| replace Original BPE with Scaffold-BPE | 1.2B language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 60.86 -> 62.26; 47.25 -> 48.07; 31.70 -> 32.90; 68.55 -> 69.86; 44.09 -> 45.34; 65.61 -> 67.02; 55.52 -> 56.00 | Helped |
| replace Original BPE with Scaffold-BPE | 6.7B language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 62.87 -> 64.95; 60.57 -> 61.19; 35.10 -> 38.00; 73.69 -> 74.54; 46.98 -> 47.49; 71.43 -> 72.26; 60.97 -> 61.76 | Helped |
| replace Original BPE with Scaffold-BPE | 6.7B language model · TriviaQA and WebQuestions | exact match: 15.63 -> 18.86; 8.56 -> 9.89 | Helped |
| replace Original BPE with Scaffold-BPE at 64K vocabulary | 468M language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 58.01 -> 59.71; 41.82 -> 42.06; 30.90 -> 31.10; 67.95 -> 69.26; 43.47 -> 43.86; 64.19 -> 65.02; 53.67 -> 54.34 | Helped |
| replace Original BPE with Scaffold-BPE at 128K vocabulary | 468M language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 56.67 -> 59.43; 42.70 -> 42.91; 31.10 -> 32.30; 67.68 -> 68.82; 43.83 -> 44.14; 64.11 -> 65.26; 53.91 -> 55.09 | Helped |
| replace Original BPE with Scaffold-BPE | 468M language model trained for 300B tokens · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 58.21 -> 61.62; 45.10 -> 46.03; 31.10 -> 33.50; 68.63 -> 69.86; 43.78 -> 44.73; 65.53 -> 66.54; 53.67 -> 56.12 | Helped |
| replace Original BPE with Scaffold-BPE using the same 388 GiB training text | 468M language model · Common-sense reasoning: BoolQ, HellaSwag, OpenBookQA, PIQA, SIQA, StoryCloze, Winogrande | accuracy: 58.72 -> 60.55; 40.84 -> 41.69; 30.55 -> 32.22; 66.58 -> 68.78; 43.40 -> 44.13; 62.85 -> 63.08; 53.07 -> 54.25 | Helped |
| combine Scaffold-BPE with BPE-Dropout | BPE tokenizer · WMT'14 En-De and En-Fr | BLEU: 29.76 -> 29.78; 43.81 -> 43.83 | Helped |
| replace Original BPE with Scaffold-BPE | tokenizer · Pile, En-De, En-Fr | compression rate: 3.879 -> 3.889; 4.830 -> 4.861; 5.012 -> 5.042 | Helped |
| replace Original BPE with Scaffold-BPE | tokenizer · Pile token representations | entropy: 11.2382 -> 11.2443 | Helped |
Surprising findings
- Scaffold tokens make up about 6.07% of the 32K vocabulary, and replacing them raises average token frequency by 76.40% at 32K vocabulary size.
Limitations
- Gains on language modeling are modest in absolute terms (often 1-3 points) and evaluated with a single pretraining run per setting.
- Main comparisons use equal training tokens, so models see different amounts of text; the equal-text-size experiment is a smaller check.
- Evaluation relies on multiple-choice and closed-book QA benchmarks; no analysis of generation quality is reported.
Experimental setup
Datasets: The Pile WMT'14 En-De WMT'14 En-Fr
Benchmarks: BoolQ HellaSwag OpenBookQA PIQA SIQA StoryCloze Winogrande TriviaQA WebQuestions WMT'14 En-De WMT'14 En-Fr
Metrics: accuracy exact match BLEU compression rate Shannon entropy redundancy p-value (t-test)
Baselines: Original BPE BPE-Dropout
Builds on: byte pair encoding BPE-Dropout Pythia LLaMA
Closest papers by meaning
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.87)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.87)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.87)
- Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.86)
- A Partition Cover Approach to Tokenization (NeurIPS 2025 · similarity 0.86)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.