Research map / Large language models / Language, safety and interpretability
Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decouples input and output vocabularies: Over-Encoding adds large hashed n-gram input embeddings while Over-Decoding/MTP predicts multi-token outputs, combined as Over-Tokenized Transformer, showing log-linear loss gains with input vocabulary size.
Problem
Tokenization's role in LLM scaling is underexplored; input and output vocabulary sizes are usually coupled, and larger output vocabularies can hurt small models, so it is unclear how vocabulary size should be scaled.
Approach
Scale only the input vocabulary via hierarchical hashed n-gram embeddings (Over-Encoding) with low-rank slicing and tiled modulo tables, while treating multi-token prediction as a form of output over-decoding; combine both into Over-Tokenized Transformer (OT).
Key results
- OE-12.8M with a 400M dense model matches the training loss of the 1B OLMo2 baseline after 400B tokens.
- On OLMoE-1.3B, OE-12.8M lowers training loss from 2.554 to 2.472 over 500B tokens; the paper reports a log-linear fit L = 2.6754 - 0.0256 log10(m).
- OT-12.8M (OE plus MTP-DS) improves OLMoE-1.3B downstream average by 1.3 points over OE-12.8M (0.537 vs 0.524), at a slight loss cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OE-12.8M | OLMoE-1.3B · OLMoE-1.3B, 500B tokens | Loss: 2.554 -> 2.472 | Helped |
| Add OE-12.8M | OLMoE-7B · OLMoE-7B, 500B tokens | Downstream average: 0.601 -> 0.608 | Helped |
| Add MTP | OLMoE-1.3B · OLMoE-1.3B, 500B tokens | Downstream average: 0.510 -> 0.508 | Hurt |
| Combine OE-12.8M with MTP-DS | OLMoE-1.3B · OLMoE-1.3B, 500B tokens | Downstream average: 0.524 -> 0.537 | Helped |
| Add a 2-gram input embedding (ablation) | OLMoE-1.3B · OLMoE-1.3B, 50B tokens | Loss: 2.714 -> 2.678 | Helped |
| Remove the 1-gram input embedding (ablation) | OLMoE-1.3B over-encoding · OLMoE-1.3B, 50B tokens | Loss: 2.678 -> 2.785 | Hurt |
| Add 3-gram inputs to the 1-gram and 2-gram hierarchy (ablation) | OLMoE-1.3B over-encoding · OLMoE-1.3B, 50B tokens | Loss: 2.678 -> 2.667 | Helped |
| Use sliced embedding tables (k=4) (ablation) | OLMoE-1.3B over-encoding · OLMoE-1.3B, 50B tokens | Loss: 2.678 -> 2.670 | Helped |
| Use 3.2M rather than 64V embedding rows (ablation) | OLMoE-1.3B over-encoding · OLMoE-1.3B, 50B tokens | Loss: 2.702 -> 2.678 | Helped |
| Add over-decoding with λ₂=0.1 (ablation) | OLMoE-1.3B · OLMoE-1.3B, 500B tokens | Eval Loss: 2.924 -> 2.920 | Helped |
| Set over-decoding loss weight λ₂=0.2 (ablation) | OLMoE-1.3B · OLMoE-1.3B, 500B tokens | Eval Loss: 2.924 -> 2.918 | Helped |
| Increase input vocabulary size m (ablation) | OLMoE-1.3B over-encoding · OLMoE-1.3B, 500B tokens | Training loss: for every 4× increase in m, the training loss decreases by 0.015 | Helped |
Surprising findings
- Larger output vocabularies (over-decoding) hurt small models in the CFG experiments, while larger input vocabularies consistently helped.
- On OLMoE-7B, OE's downstream gain is small (+0.007) despite a loss improvement of -0.076, which the authors attribute to overlap with MoE sparse parameters.
Limitations
- Gains on MoE models in downstream metrics diminish, with the authors hypothesizing overlap with sparse MoE parameters.
- OD/MTP benefits only appear for sufficiently large models and after long training; the 7B OE gains had not converged at 500B tokens.
- Training overhead depends on engineering (tensor-parallel embedding sharding); the 7B setting was not carefully optimized.
Experimental setup
Datasets: c4_en validation context-free grammar synthetic data (cfg3f) OLMo2 pretraining data OLMoE pretraining data in-house pretraining data
Benchmarks: MMLU-Var Hellaswag ARC-Challenge ARC-Easy PIQA Winogrande Social IQA MMLU C-Eval TriviaQA MMLU-Pro AGIEval
Metrics: training loss eval loss perplexity downstream accuracy training throughput prefill and decode tokens per second generation accuracy (CFG)
Baselines: OLMo2-1B OLMoE-1.3B OLMoE-7B MTP MTP-DS BLT
Builds on: OLMo2 OLMoE Multi-Token Prediction DeepSeek-V3 MTP FSDP
Closest papers by meaning
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.93)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.91)
- Model-Aware Tokenizer Transfer (NeurIPS 2026 · similarity 0.90)
- From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.90)
- Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.90)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.