Research map / Large language models / Language, safety and interpretability
Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces data mixture inference, a linear-programming attack that recovers the language, code, and domain proportions of a BPE tokenizer's training data from its ordered merge list.
Problem
Pretraining data mixtures of modern LMs are undisclosed, and membership inference attacks target instances or authors rather than global corpus proportions.
Approach
The ordered BPE merge list reveals token-pair frequencies in the training data: each merge must be the most frequent pair at its step. Given sample corpora per category, the attack formulates a relaxed linear program whose constraints encode these inequalities and solves for category proportions, using lazy constraint and variable generation to scale.
Key results
- On controlled mixtures, the attack achieves mean log10 MSE of about -7.7 for 112 languages versus -3.8 for random guessing, and -4.5 for the best baseline (TC).
- Inferred GPT-4o training data is 39% non-English, versus 3.2% for GPT-3.5, and GPT-3.5 is about 62.6% code.
- Llama 3's extension of GPT-3.5's merge list (27,744 merges) is inferred to be 48.5% non-English and 30.2% code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ordered BPE merge constraints instead of random guessing | Data mixture inference · Natural language mixtures (n=5) | log10(MSE): −1.39±0.36 -> −7.30±1.31 | Helped |
| Use ordered BPE merge constraints instead of tokenizer encoding efficiency | Data mixture inference · Natural language mixtures (n=5) | log10(MSE): −2.02±0.41 -> −7.30±1.31 | Helped |
| Use ordered BPE merge constraints instead of token classification | Data mixture inference · Natural language mixtures (n=5) | log10(MSE): −2.12±0.49 -> −7.30±1.31 | Helped |
| Use ordered BPE merge constraints instead of tokenizer encoding efficiency | Data mixture inference · Programming language mixtures (n=5) | log10(MSE): −2.54±0.42 -> −6.46±0.79 | Helped |
| Use ordered BPE merge constraints instead of token classification | Data mixture inference · Programming language mixtures (n=5) | log10(MSE): −1.92±0.36 -> −6.46±0.79 | Helped |
| Use ordered BPE merge constraints instead of tokenizer encoding efficiency | Data mixture inference · Domain mixtures (n=5) | log10(MSE): −1.69±0.29 -> −3.74±0.94 | Helped |
| Use ordered BPE merge constraints instead of token classification | Data mixture inference · Domain mixtures (n=5) | log10(MSE): −1.64±0.35 -> −3.74±0.94 | Helped |
| Estimate merge frequencies from Wikipedia rather than the tokenizer training domain (ablation) | Data mixture inference · 10-language mixtures | log MSE: −7.66 -> −3.53 | Hurt |
| Use character-level rather than byte-level BPE (ablation) | Data mixture inference · 10-language mixtures | log MSE: −7.65 -> −4.09 | Hurt |
| Increase sampled data per category for merge-frequency estimates (ablation) | Data mixture inference · Mixtures of n=5 categories | MSE | Helped |
| Increase the number of merges considered (ablation) | Data mixture inference · Controlled mixtures | MSE | Helped |
| Omit categories from the inference solver (ablation) | Data mixture inference · 112-language mixtures | MSE | Hurt |
Surprising findings
- Character-level sentencepiece tokenizers are harder to attack than byte-level ones (log MSE -4.09 vs -7.65).
- All studied tokenizers appear to include 7%-26% book data, possibly because books use a more standard vocabulary than the web.
Limitations
- Performance degrades under distribution shift between tokenizer training data and attacker's sample data (log MSE -3.53 vs -7.66).
- Requires a reasonable reconstruction of pretokenization rules and a complete category set; unaccounted-for categories reduce accuracy.
- Inferred proportions for commercial tokenizers are estimates of the tokenizer's training data, which may differ from the LM's pretraining data.
Experimental setup
Datasets: OSCAR v23.01 RedPajama (GitHub, Wikipedia, Common Crawl, Books, ArXiv splits)
Benchmarks: Controlled mixtures of 112 natural languages, 37 programming languages, and 5 domains Commercial tokenizers of GPT, Llama, Mistral, GPT-NeoX, Claude, Gemma families
Metrics: log10 MSE of estimated mixture proportions
Baselines: Random guessing Tokenizer Encoding Efficiency (TEE) Token Classification (TC)
Builds on: byte-pair encoding (Sennrich et al.) membership inference attacks distribution inference attacks
Closest papers by meaning
- Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.89)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.88)
- Aioli: A Unified Optimization Framework for Language Model Data Mixing (ICLR 2025 · similarity 0.88)
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.