Research map / Architectures and efficiency / Attention and transformers
MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MrT5, a ByT5 variant whose encoder has a learned delete gate at an early layer that dynamically removes tokens, cutting sequence length by up to 75% with comparable accuracy and faster inference.
Problem
Byte-level language models like ByT5 avoid subword tokenization drawbacks but produce much longer sequences, making training and inference slow and inefficient.
Approach
A learned, fully differentiable delete gate after a fixed early encoder layer soft-masks tokens during training and hard-deletes them at inference, so remaining tokens absorb context from deleted ones; a PI controller can target a specific deletion ratio, and the gate adds only 2d_model+1 parameters to a pre-trained ByT5.
Key results
- On XNLI, MrT5 reaches 65.31% accuracy vs 64.72% for ByT5 with 55.60% sequence length reduction and 45.13% runtime decrease.
- On character-level tasks, MrT5 cuts sequence length by 74.53% on word search and runtime by 59.29%, with sequence-level accuracy 72.02% vs ByT5's 76.61%.
- In continued pre-training, MrT5 with delta=0.5 achieves lower BPB than random, fixed, BP and CP baselines at comparable compression, and MrT5 Small's runtime decreases 32.27% at delta=0.5 vs ByT5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a learned delete gate targeting δ=0.5 | ByT5 Small · XNLI | accuracy: 64.72 -> 65.31 | Helped |
| add a learned delete gate targeting δ=0.5 | ByT5 Small · TyDiQA-GoldP | EM/F1: 69.90/79.58 -> 68.27/77.73 | Hurt |
| add a learned delete gate targeting δ=0.5 | ByT5 Small · Spelling Correction | sequence-level accuracy: 66.73 -> 63.18 | Hurt |
| add a learned delete gate targeting δ=0.7 | ByT5 Small · Word Search | sequence-level accuracy: 76.61 -> 72.02 | Hurt |
| add a learned delete gate | ByT5 Small · XNLI | runtime decrease: 45.13% | Helped |
| increase the target deletion ratio from δ=0.3 to δ=0.7 | MrT5 · inference on a single sequence | runtime (ms): 34.99 -> 24.48 | Helped |
| move the delete gate to an earlier encoder layer (ablation) | MrT5 · continued pre-training | bits-per-byte | Hurt |
| remove Gumbel noise from the delete gate during training (ablation) | MrT5 · continued pre-training | performance | No clear effect |
| vary the deletion regularizer α (ablation) | MrT5 · Simple Vowel Removal | token-level accuracy: 100.00 -> 100.00 | No clear effect |
| vary the deletion regularizer α (ablation) | MrT5 · Contextual Vowel Removal | sequence length reduction: 1.56 -> 18.97 | Helped |
Surprising findings
- Adding a softmax_1 and the delete gate to a pre-trained ByT5 barely changed validation loss (0.7919 vs 0.7917 for unaltered ByT5), supporting the use of softmax_1.
- The 1.23B MrT5 Large showed a smaller BPB gap to ByT5 than the Small model, with a 44.6% runtime reduction at 49.5% sequence length decrease.
Limitations
- Gate placed at a single early layer; BPB is higher when placed earlier, so compute savings are bounded by gate placement.
- MrT5 generally has higher BPB than ByT5, with gap growing at higher deletion ratios.
- Downstream tasks evaluated with ByT5 Small-scale models and specific fine-tuning settings; per-task deletion ratios are tuned manually.
Experimental setup
Datasets: mC4 C4 MultiNLI XNLI TyDiQA-GoldP character-level spelling correction and word search tasks
Benchmarks: XNLI TyDiQA-GoldP Spelling Correction with Context Word Search mC4 span corruption (bits-per-byte)
Metrics: bits-per-byte accuracy exact match F1 sequence-level accuracy sequence length reduction inference runtime decrease
Baselines: ByT5 random deletion fixed deletion boundary predictor pooling convolutional pooling (CANINE-style)
Builds on: ByT5 mT5 Hourglass Transformer softmax1
Closest papers by meaning
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.89)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.88)
- Better & Faster Large Language Models via Multi-token Prediction (ICML 2024 · similarity 0.88)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.88)
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.