Research map / Architectures and efficiency / Attention and transformers
The Efficiency Gap in Byte Modeling
Celine Lee, Jing Nathan Yan, Chen Liang, Jiaxin Shi, Yin Zhang, Jeremiah Zhe Liu, Pengcheng Yin, Fernando Pereira, Ed H. Chi, Derek Zhiyuan Cheng, Alexander M Rush, Ruoxi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bytes instead of BPE tokens (compute-matched, 180M vs 717M) | AR model · Zero-shot average | accuracy: 46.23 -> 41.71 | Hurt |
| use bytes instead of BPE tokens (compute-matched) | MDM · Zero-shot average | accuracy: 40.70 -> 37.73 | Hurt |
| increase byte model capacity from 180M to 717M | AR byte · Zero-shot average | accuracy: 41.71 -> 44.23 | Improved |
| increase byte model capacity from 180M to 717M | MDM byte · Zero-shot average | accuracy: 37.73 -> 36.76 | No clear effect |
| use AR instead of MDM objective | 717M byte · Zero-shot average | accuracy: 36.76 -> 44.23 | Improved |
| compute needed for byte to match BPE: AR vs MDM | byte models · BPB compute gap | compute ratio: AR 7.9x at BPB 1.0 vs MDM 20.5x at BPB 1.0 | Hurt |
| scale compute for AR byte models | AR byte vs BPE · BPB compute gap | compute ratio: 7.9x at BPB 1.0 -> 2.3x at BPB 0.8 | Improved |
| global random permutation applied to bytes vs BPE (ablation) | AR · DEFLATE compressibility | loss (%): BPE 6.9% vs byte 19.3% | Hurt |
| preserve local contiguity (inter-block-8 vs inter-block-4) (ablation) | AR byte · DEFLATE compressibility loss | loss (%): 14.0% -> 9.0% | Improved |
| preserve global order (intra-block-8) vs inter-block-8 (ablation) | AR byte · training under permutation | model performance: 16.4% vs 9.0% compressibility loss | Improved |
| BPE-based span masking on byte MDM (ablation) | 180M byte MDM · validation BPB | BPB | Hurt |
| larger vocabularies (GPT-2, Llama-3) for BPE MDM (ablation) | MDM BPE · iso-FLOPs, 180M and 717M | BPB | Mixed |
Closest papers by meaning
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.89)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.88)
- Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models (NeurIPS 2026 · similarity 0.88)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.87)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
