AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

The Efficiency Gap in Byte Modeling

Celine Lee, Jing Nathan Yan, Chen Liang, Jiaxin Shi, Yin Zhang, Jeremiah Zhe Liu, Pengcheng Yin, Fernando Pereira, Ed H. Chi, Derek Zhiyuan Cheng, Alexander M Rush, Ruoxi Wang

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: Language model pretraining and induction · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use bytes instead of BPE tokens (compute-matched, 180M vs 717M)AR model · Zero-shot averageaccuracy: 46.23 -> 41.71Hurt
use bytes instead of BPE tokens (compute-matched)MDM · Zero-shot averageaccuracy: 40.70 -> 37.73Hurt
increase byte model capacity from 180M to 717MAR byte · Zero-shot averageaccuracy: 41.71 -> 44.23Improved
increase byte model capacity from 180M to 717MMDM byte · Zero-shot averageaccuracy: 37.73 -> 36.76No clear effect
use AR instead of MDM objective717M byte · Zero-shot averageaccuracy: 36.76 -> 44.23Improved
compute needed for byte to match BPE: AR vs MDMbyte models · BPB compute gapcompute ratio: AR 7.9x at BPB 1.0 vs MDM 20.5x at BPB 1.0Hurt
scale compute for AR byte modelsAR byte vs BPE · BPB compute gapcompute ratio: 7.9x at BPB 1.0 -> 2.3x at BPB 0.8Improved
global random permutation applied to bytes vs BPE (ablation)AR · DEFLATE compressibilityloss (%): BPE 6.9% vs byte 19.3%Hurt
preserve local contiguity (inter-block-8 vs inter-block-4) (ablation)AR byte · DEFLATE compressibility lossloss (%): 14.0% -> 9.0%Improved
preserve global order (intra-block-8) vs inter-block-8 (ablation)AR byte · training under permutationmodel performance: 16.4% vs 9.0% compressibility lossImproved
BPE-based span masking on byte MDM (ablation)180M byte MDM · validation BPBBPBHurt
larger vocabularies (GPT-2, Llama-3) for BPE MDM (ablation)MDM BPE · iso-FLOPs, 180M and 717MBPBMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy