Research map / Architectures and efficiency / RNNs and normalization

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training

NeurIPS 2025 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled study scaling BPE vocabulary from 24K to 196K with fixed data, compute and optimization, showing larger vocabularies lower cross-entropy mainly by reducing loss on the ~2,500 most frequent words while rare-tail loss rises.

Problem

Why enlarging the tokenizer vocabulary improves language model performance, and which part of the loss (frequent vs. rare words) drives the gain.

Approach

Frames vocabulary scaling through Kolmogorov complexity of tokenized text and token-frequency imbalance; shows that above ~24K all common words are already single tokens, so further growth mainly sharpens frequency skew, and the global cross-entropy gain comes almost entirely from frequent-word loss. Also shows parameter scaling with fixed vocabulary yields the same frequent-word benefit.

Key results

• Enlarging vocabulary from 24K to 196K lowers global cross-entropy from about 3.179 to 3.136 nats on an 85M model trained on 10B FineWeb-Edu tokens, with frequent-word loss dropping ~0.1 nats while rare-word loss rises (11.183 to 13.399 for the rarest 20K words).
• Kolmogorov complexity upper bound on the 45.97B-byte FineWeb-Edu corpus decreases from 10.74B (24K) to 10.16B (196K) bytes, and NCR from 0.234 to 0.221.
• Segmentation efficiency exceeds 95% by 24K, so beyond that vocabulary growth mainly increases token-frequency imbalance (JSD grows monotonically).

Measured effects reported in the paper

Surprising findings

• Rare-word loss rises sharply with vocabulary size (e.g., 11.183 to 13.399 nats for the rarest 20K words), yet global cross-entropy still falls.
• A size-matched 49K-vocabulary model (161M params) with the 24K baseline's hidden dims is not strictly better, but parameter scaling with a fixed vocabulary reproduces the frequent-word gain.

Limitations

• Kolmogorov complexity is approximated by unigram Shannon entropy times token count, which is a loose upper bound.
• Frequency-based mechanism is argued from correlations and overlap analysis; causal claims rest on limited model sizes (85M, 450M) and token budgets.
• Machine translation results contrast with the monolingual findings and are only hypothesized to be due to rare-word issues.

Experimental setup

Datasets: FineWeb-Edu OpenWebText Paloma ARC HellaSwag SciQ PIQA CC-Main-2023-40 Google Web Trillion Word Corpus

Benchmarks: ARC-Easy HellaSwag PIQA SciQ

Metrics: cross-entropy loss average per-word loss frequent-word loss rare-word loss Jensen-Shannon divergence normalized compression ratio downstream accuracy

Baselines: SuperBPE Pythia OLMo-2

Builds on: BPE SuperBPE pre-LN AdamW

Closest papers by meaning

• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.91)
• Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.90)
• Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.89)
• From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.88)
• Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.88)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.88)