Research map / Probabilistic and scientific modeling / Molecules and proteins
Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MxDNA, which learns DNA tokenization end-to-end via a sparse Mixture of Convolution Experts plus deformable convolution inside a Nucleotide Transformer v2 100M backbone, pretrained with masked language modeling on the human reference genome.
Problem
Existing DNA foundation models borrow NLP tokenizers (single nucleotide, k-mer, BPE) that are ill-suited to DNA, whose meaningful units are discontinuous, overlapping, and ambiguous, and the optimal DNA tokenization is unknown.
Approach
Learn tokenization via gradient descent: a sparse mixture of variable-width convolution experts proposes basic units (with NMS and jitter noise for ambiguity), and a 1-D deformable convolution assembles them into overlapping, discontinuous final tokens, with cross-attention restoring nucleotide resolution for MLM pretraining.
Key results
- On Nucleotide Transformer Benchmarks MxDNA averages 78.14 vs 76.66 for DNABERT2 (second best), best on 10 of 18 datasets.
- On Genomic Benchmarks MxDNA averages 89.13 vs 88.29 for DNABERT2, best on 5 of 8 datasets.
- Ablation: learnt tokenization (77.52 NT avg) outperforms single-nucleotide (75.07), overlapping 6-mer (74.35), non-overlapping 6-mer (67.65), and BPE (74.96).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MxDNA instead of DNABERT2 | DNA foundation model · Genomic Benchmarks | average accuracy: 88.29 -> 89.13 | Helped |
| Use MxDNA instead of DNABERT2 | DNA foundation model · Nucleotide Transformer Benchmarks | average score: 76.66 -> 78.14 | Helped |
| Replace rule-based tokenization with learnt tokenization (ablation) | same-backbone DNA model · Genomic Benchmarks | average score: 88.56 -> 88.89 | Helped |
| Replace rule-based tokenization with learnt tokenization (ablation) | same-backbone DNA model · Nucleotide Transformer Benchmarks | average score: 75.07 -> 77.52 | Helped |
| Add Mixture of Convolution Experts (ablation) | single-nucleotide baseline · Genomic Benchmarks | average score: 88.56 -> 88.72 | Helped |
| Add Mixture of Convolution Experts (ablation) | single-nucleotide baseline · Nucleotide Transformer Benchmarks | average score: 75.07 -> 77.00 | Helped |
| Add deformable convolution (ablation) | model with Mixture of Convolution Experts · Genomic Benchmarks | average score: 88.72 -> 88.86 | Helped |
| Add deformable convolution (ablation) | model with Mixture of Convolution Experts · Nucleotide Transformer Benchmarks | average score: 77.00 -> 77.35 | Helped |
| Add multiplicative jitter noise (ablation) | model with deformable convolution · Genomic Benchmarks | average score: 88.86 -> 88.89 | Helped |
| Add multiplicative jitter noise (ablation) | model with deformable convolution · Nucleotide Transformer Benchmarks | average score: 77.35 -> 77.52 | Helped |
Limitations
- Direct biological validation of learned tokenization decisions is limited.
- Evaluation on long-range tasks is lacking due to quadratic self-attention cost.
- Baselines were reproduced under uniform hyperparameters and HyenaDNA results were somewhat below originally reported values.
Experimental setup
Datasets: Human Reference Genome Nucleotide Transformer Benchmarks Genomic Benchmarks
Benchmarks: Nucleotide Transformer Benchmarks Genomic Benchmarks
Metrics: MCC F1 top-1 accuracy
Baselines: Nucleotide Transformer v2 100M DNABERT DNABERT2 HyenaDNA single nucleotide tokenization overlapping 6-mer non-overlapping 6-mer BPE
Builds on: Nucleotide Transformer v2 Mixture of Experts Deformable convolution DNABERT2
Closest papers by meaning
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.87)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.86)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.86)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.86)
- VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling (ICML 2024 · similarity 0.86)
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.