Research map / Large language models / Language, safety and interpretability

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that optimal LLM vocabulary size grows with compute budget and proposes three approaches (IsoFLOPs power-law fits, derivative-based estimation, and a parametric loss fit) to predict it, validated with models up to 3B parameters.

Problem

Scaling laws for LLMs ignore vocabulary size, leading to widely varying and likely suboptimal vocabulary sizes; the compute-optimal vocabulary size for a given model and budget is unknown.

Approach

Introduces a vocabulary-insensitive unigram-normalized loss and a tokens-per-character function f(V), then predicts optimal vocabulary via IsoFLOPs power-law fits, a derivative of FLOPs w.r.t. V, and a parametric loss with a vocabulary term.

Key results

• Predicted optimal vocabulary for Llama2-70B is at least 216K, about 7x the 32K vocabulary.
• At 2.3e21 FLOPs, increasing vocabulary from 32K to 43K raised ARC-Challenge from 29.1 to 32.0.
• Fitted power laws give N_v ∝ C^0.42 and N_nv ∝ C^0.50, so vocabulary parameters scale slower than non-vocabulary parameters (gamma ≈ 0.83).

Measured effects reported in the paper

Surprising findings

• Optimal vocabulary size decreases when training data is the bottleneck and increases under overtraining, e.g. Llama3-8B-style excess data.
• Most current LLMs (e.g., Llama2) allocate fewer vocabulary parameters than the predicted optimum, despite a common assumption of 32K being sufficient.

Limitations

• Approach 2 relies on the accuracy of the FLOPs equation and the tokens-characters function and cannot independently allocate non-vocabulary parameters and data.
• Experiments only cover up to 3B parameters and vocabulary sizes 4K-96K, so extrapolation to 70B+ models is predictive rather than verified.
• Analysis focuses on English pre-training corpora and training-compute-optimal settings; inference cost of larger vocabularies is noted but not fully analyzed.

Experimental setup

Datasets: SlimPajama

Benchmarks: ARC-Challenge ARC-Easy HellaSwag OpenBookQA WinoGrande PIQA BoolQ

Metrics: accuracy normalized loss (L_u) RMSE R^2

Baselines: Llama2 vocabulary 32K Gemma vocabulary configurations

Builds on: Chinchilla scaling laws unigram-normalized metric Llama architecture

Closest papers by meaning

• LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.91)
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.90)
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.90)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
• Position: Enough of Scaling LLMs! Lets Focus on Downscaling (ICML 2025 · similarity 0.89)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)