Research map / Large language models / Language, safety and interpretability
BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BEND, a benchmark of seven biologically meaningful human-genome tasks for DNA language models, and evaluates frozen embeddings from about a dozen DNA LMs with lightweight CNN heads against supervised and expert baselines.
Problem
Existing DNA language models are evaluated on inconsistent, short-context tasks that do not capture long-range genome organization, sparsity, and scale, making fair comparison and progress measurement difficult.
Approach
A standardized benchmark of seven human-genome tasks spanning length scales (gene finding, enhancer annotation, chromatin accessibility, histone modification, CpG methylation, and two zero-shot noncoding variant effect tasks), evaluated by training a lightweight two-layer CNN on frozen LM embeddings and by zero-shot cosine distance for variant effects.
Key results
- NT-MS reaches 0.68 MCC on gene finding, outperforming other LMs but below AUGUSTUS (0.80).
- DNABERT achieves 0.85 AUROC on chromatin accessibility, on par with Basset (0.85).
- On enhancer annotation all models perform poorly (AUPRC at most 0.06) versus Enformer at 0.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NT-MS embeddings with the downstream CNN instead of the one-hot ResNet | Gene finding · Gene finding | MCC: 0.46 -> 0.68 | Helped |
| Use NT-MS embeddings with the downstream CNN instead of the one-hot ResNet | Enhancer annotation · Enhancer annotation | AUPRC: 0.06 -> 0.06 | No clear effect |
| Use DNABERT embeddings instead of one-hot input | Chromatin accessibility prediction · Chromatin accessibility | AUROC: 0.75 -> 0.85 | Helped |
| Use DNABERT embeddings instead of Basset | Histone modification prediction · Histone modification | AUROC: 0.74 -> 0.79 | Helped |
| Use NT-MS embeddings instead of Basset | CpG methylation prediction · CpG methylation | AUROC: 0.93 -> 0.92 | Hurt |
| Use DNABERT for zero-shot variant effect prediction instead of DeepSEA | Variant effect prediction · Variant effects (expression) | AUROC: 0.70 -> 0.60 | Hurt |
| Use NT-MS for zero-shot variant effect prediction instead of DeepSEA | Variant effect prediction · Variant effects (disease) | AUROC: 0.56 -> 0.77 | Helped |
Surprising findings
- A two-layer CNN on one-hot DNA fails to learn gene finding (MCC 0.00), while LM embeddings enable non-trivial performance.
- The much smaller DNABERT outperforms larger NT-MS on some tasks, such as chromatin accessibility and expression variant effects.
Limitations
- Restricted to the human genome, so cross-organism generalization is untested.
- Frozen embeddings with a lightweight CNN head may conflate representation quality with downstream architecture inductive bias.
- Enhancer annotation dataset is small (285 genes), limiting statistical power.
Experimental setup
Datasets: GENCODE ENCODE DNase-I hypersensitivity ENCODE histone ChIP-seq ENCODE WGBS methylation CRISPRi enhancer-gene pairs DeepSEA eQTL SNPs ClinVar 1000 Genomes Project
Benchmarks: BEND
Metrics: MCC AUPRC AUROC
Baselines: AUGUSTUS Enformer Basset DeepSEA ResNet CNN (one-hot) two-layer CNN (one-hot)
Builds on: DNABERT Nucleotide Transformer GUE Genomic Benchmarks Basset
Closest papers by meaning
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (NeurIPS 2024 · similarity 0.89)
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.87)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.87)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.87)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.86)
- What DNA Foundation Models Learn Beyond Sequence Composition (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.