Research map / Architectures and efficiency / Attention and transformers
MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAGNET, a byte-level multilingual LM whose gradient-based tokenizer uses script-specific boundary predictors with script-dependent binomial priors, reducing over-segmentation of non-Latin scripts while keeping downstream performance.
Problem
Tokenization in multilingual LMs over-segments non-Latin script and low-resource languages, causing unequal utility, efficiency, and cost; existing gradient-based and dynamic pooling tokenizers apply one compression rate to all scripts.
Approach
Route byte sequences through language-script-specific boundary predictors (hourglass transformer with Gumbel-sigmoid boundaries), each with a binomial prior set from the anchor language's byte-to-word ratio, so that segmentation granularity is equitable across scripts.
Key results
- MAGNET (3,6,12)x reaches 85.41 on PAWS-X vs 82.18 for byte-level and 81.29 for DTP 5x; MAGNET (1,2,4)x reaches 68.74 on XNLI vs 68.68 for byte-level.
- MAGNET cuts Indic-language token counts by roughly 5x relative to DTP and BPE subword tokenizers on FLORES, giving more equitable segmentation across scripts.
- Inference time relative to byte-level is about 0.29x to 0.38x for MAGNET on XQuAD, versus 0.38x for the byte-level baseline in English and Russian.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use script-specific predictors with MAGNET (1, 2, 4)× | Byte-level multilingual model · XNLI | accuracy: 68.68 -> 68.74 | Helped |
| use script-specific predictors with MAGNET (3, 6, 12)× | Byte-level multilingual model · PAWS-X | accuracy: 82.18 -> 85.41 | Helped |
| use script-specific predictors with MAGNET (3, 6, 12)× | Byte-level multilingual model · SIB-200 | accuracy: 71.05 -> 71.43 | Helped |
| use script-specific predictors with MAGNET (1, 2, 4)× | Byte-level multilingual model · XQUAD | accuracy: 44.62 -> 44.61 | No clear effect |
| use script-specific predictors with MAGNET | Byte-level multilingual model · Hascova | accuracy: 87.38 -> 87.62 | No clear effect |
| use script-specific predictors with MAGNET | Byte-level multilingual model · ILI | accuracy: 89.24 -> 88.72 | No clear effect |
| increase compression to (5, 10, 20)× (ablation) | MAGNET · XNLI | accuracy: 68.74 -> 67.50 | Hurt |
| increase compression to (5, 10, 20)× (ablation) | MAGNET · PAWS-X | accuracy: 83.30 -> 80.44 | Hurt |
| increase compression to (5, 10, 20)× (ablation) | MAGNET · XQUAD | accuracy: 44.61 -> 41.06 | Hurt |
| use (3, 6, 12)× instead of (1, 2, 4)× compression (ablation) | MAGNET · PAWS-X | accuracy: 83.30 -> 85.41 | Helped |
| use (3, 6, 12)× instead of (1, 2, 4)× compression (ablation) | MAGNET · XQUAD | accuracy: 44.61 -> 43.72 | Hurt |
| reduce Indic compression to (5, 10, 13)× (ablation) | MAGNET · SIB-200 | accuracy: 71.02 -> 70.64 | Hurt |
Surprising findings
- Fine-tuning does not noticeably change learned segmentation boundaries, despite boundary predictors being updated.
- Alpha-sampled BPE with large vocabularies still shows pronounced segmentation disparity for Indic languages, contrary to prior claims that alpha sampling alleviates disparity.
Limitations
- Limited to nine languages, a small pretraining subset, and ~126M-parameter models due to resource constraints.
- High compression hurts some languages, notably Indic languages (about 5% average drop as compression increases).
- Byte-to-word ratio proxy is hard to compute for CJK and Semitic languages that lack whitespace word boundaries.
Experimental setup
Datasets: OSCAR FLORES-200 XQuAD XNLI PAWS-X SIB-200 ILI HaSCoSVa-2022
Benchmarks: XNLI PAWS-X SIB-200 XQuAD ILI HaSCoSVa-2022 FLORES-200
Metrics: accuracy average number of tokens per sentence relative inference time
Baselines: byte-level model DTP 5x DTP 10x BPE 50k BPE 100k BPE 250k
Builds on: hourglass transformer dynamic token pooling Gumbel-sigmoid byte-level BPE
Closest papers by meaning
- Explaining and Mitigating Crosslingual Tokenizer Inequities (NeurIPS 2025 · similarity 0.88)
- MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025 · similarity 0.88)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.88)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.87)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.87)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.