Research map / Probabilistic and scientific modeling / Molecules and proteins
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces k-mer tokenization with BPE in a multi-species genome foundation model (DNABERT-2) and introduces the GUE benchmark of 36 datasets across 9 tasks, matching NT-2500M-multi with 21x fewer parameters.
Problem
k-mer tokenization in genome language models causes information leakage, poor sample efficiency, and high computational cost, and existing genome benchmarks are non-standardized and either too easy or too hard.
Approach
Adapt SentencePiece BPE tokenization (non-overlapping, variable-length tokens) for DNA, combined with ALiBi, FlashAttention, GEGLU and an MLM objective pretrained on multi-species genomes, plus a calibrated multi-species classification benchmark (GUE/GUE+).
Key results
- DNABERT-2 averages 66.80 on GUE versus 66.93 for NT-2500M-multi, with 117M vs 2537M parameters and about 19x fewer FLOPs.
- DNABERT-2 outperforms DNABERT on 23 of 28 GUE datasets, with about 6 points higher average score.
- On GUE+, DNABERT-2 beats DNABERT and NT-2500M-multi on all datasets, e.g. 48.50 vs 45.00 on virus species classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace K-mer tokenization with BPE (ablation) | DNABERT-2 · GUE (28 datasets) | average score: 60.92 -> 65.33 | Helped |
| Further pre-train on the GUE training sets (ablation) | DNABERT-2 · GUE (28 datasets) | average score: 66.80 -> 67.77 | Helped |
| Further pre-train on GUE training sets (ablation) | DNABERT-2 · GUE tasks | task score | Mixed |
| Replace BPE tokenization with K-mer tokenization (ablation) | DNABERT-2 · GUE (28 datasets) | average score: 65.33 -> 60.92 | Hurt |
Surprising findings
- Larger BPE vocabularies reduce compute but do not consistently improve performance; the 4096-token vocabulary was chosen as the balance point.
- DNABERT variants excel on 70-bp core promoter detection but degrade on 300-bp promoter detection, suggesting shorter tokenized sequences lose subtle signals.
Limitations
- Evaluation is limited to classification tasks; generative or regression genome tasks are not covered.
- Hyperparameters are kept mostly fixed across datasets, so per-dataset tuning is not explored.
- Non-overlapping BPE shortens sequences, which can hurt detection of subtle short-signal tasks such as promoter detection at 300 bp.
Experimental setup
Datasets: Multi-species genome (135 species) Human genome (DNABERT pretraining set) GUE GUE+ Epigenetic marks data (Nucleotide Transformer)
Benchmarks: GUE GUE+
Metrics: F1-score Matthews Correlation Coefficient parameters FLOPs
Baselines: DNABERT (3/4/5/6-mer) Nucleotide Transformer (NT-500M-human) NT-500M-1000g NT-2500M-1000g NT-2500M-multi HyenaDNA CNN
Builds on: DNABERT SentencePiece Byte Pair Encoding ALiBi FlashAttention
Closest papers by meaning
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.87)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.87)
- VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling (ICML 2024 · similarity 0.87)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.87)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.86)
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.