Research map / Large language models / Reasoning and chain of thought
Enhancing Large Language Models through Adaptive Tokenizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ADAT, a tokenizer learned by iteratively pruning a large initial vocabulary using a Unigram-style loss combined with per-token cross-entropy from a Pythia model trained on the vocabulary.
Problem
Traditional subword tokenizers (BPE, Unigram, BytePiece) are static, frequency-based, and not synchronized with the LLM being trained, which may limit model performance.
Approach
Iteratively prune a large initial vocabulary (150k from BytePiece or Unigram) by combining frequency-based Unigram loss with per-token cross-entropy loss from a randomly initialized LLM trained on the current vocabulary, using randomized Viterbi sampling and loss momentum.
Key results
- ADAT on Unigram initialization reaches 44.51 average accuracy vs 42.22 for Unigram and 41.91 for BPE at the 70M scale (Pythia-70M).
- At 70M, PPL on PG19 drops from 16.52 (Unigram) to 6.97 with ADAT; at 410M, PPL drops from 10.92 to 5.78.
- Gains persist at 160M (44.04 to 46.05 average) and 410M (46.50 to 48.42 average); a 1B model on 60B tokens shows 49.11 to 51.20 average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adaptively refine the Unigram vocabulary with ADAT | Pythia-70M · Eight-benchmark average | Avg. (%): 42.22 -> 44.51 | Helped |
| Adaptively refine the Unigram vocabulary with ADAT | Pythia-70M · PG19 test | PPL: 16.52 -> 6.97 | Helped |
| Adaptively refine the BytePiece vocabulary with ADAT | Pythia-70M · Eight-benchmark average | Avg. (%): 37.21 -> 39.91(+2.70) | Helped |
| Replace Unigram tokenizer with ADAT | Pythia-160M · Eight-benchmark average | Avg: 44.04 -> 46.05 | Helped |
| Replace Unigram tokenizer with ADAT | Pythia-410M · Eight-benchmark average | Avg: 46.50 -> 48.42 | Helped |
| Replace Unigram tokenizer with ADAT | Pythia-1B · 60B corpus | Avg: 49.11 -> 51.20 | Helped |
| Use 100M inference tokens for token-loss calculation rather than 1M (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 43.13 -> 44.51 | Helped |
| Increase inference data volume from 10M to 100M tokens (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 43.74 -> 44.51 | Helped |
| Use 150K rather than 75K initial vocabulary (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 43.42 -> 44.51 | Helped |
| Use 150K rather than 100K initial vocabulary (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 43.78 -> 44.51 | Helped |
| Add loss momentum to vocabulary pruning (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 43.16 -> 44.51 | Helped |
| Use a − λb balance function instead of a/λ log(b + 1) (ablation) | ADAT on Pythia-70M · Tokenizer accuracy | Accuracy: 44.51 -> 42.70 | Hurt |
Surprising findings
- The ADAT vocabulary overlaps only about 18-21% with the Unigram vocabulary of the same size, indicating substantially different tokens.
- ADAT's compression rate is lower than Unigram's (2.91 vs 3.96 for the ADAT-Unigram variant), so the gains do not come from better compression.
Limitations
- Adaptive tokenizers produce model-specific vocabularies that are inconsistent across model sizes, complicating knowledge distillation and speculative decoding.
- Evaluated only on small Pythia models (70M-410M, one 1B run) with few-shot benchmarks; the random-initialization ablation and reported gains are modest on some tasks.
- Code and data are not yet released, limiting reproducibility.
Experimental setup
Datasets: The Pile (subset)
Benchmarks: PG19 perplexity Lambada PIQA WinoGrande ARC-Easy ARC-Challenge SciQ LogiQA SST-2 BoolQ
Metrics: perplexity accuracy average accuracy compression rate
Baselines: BPE Unigram BytePiece
Builds on: Unigram language model tokenization Byte Pair Encoding Pythia BytePiece
Closest papers by meaning
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.93)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.92)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.91)
- Getting the most out of your tokenizer for pre-training and domain adaptation (ICML 2024 · similarity 0.89)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.89)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.