Research map / Large language models / Language, safety and interpretability
DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DART-Eval, a benchmark suite of five regulatory-DNA tasks that evaluates six DNA language models (zero-shot, probed, LoRA fine-tuned) against ab initio CNN baselines such as ChromBPNet.
Problem
Existing DNA language model (DNALM) benchmarks use surrogate tasks, compositionally uncontrolled negatives, outdated baselines, and do not test regulatory motif discovery, quantitative activity prediction, or counterfactual variant effects.
Approach
A five-task benchmark for regulatory DNA with compositionally matched (dinucleotide-shuffled) negatives, 1,443 HOCOMOCO motif sensitivity tests, cell-type-specific ATAC-seq classification, and caQTL/dsQTL variant scoring, comparing zero-shot, probed and LoRA fine-tuned DNALMs against ab initio CNNs including ChromBPNet.
Key results
- Fine-tuned DNALMs reach paired accuracies approaching 1 on cCRE vs shuffled-control discrimination, matching or moderately beating the ab initio CNN (0.973 vs 0.932 paired for best fine-tuned model vs probing-head CNN).
- On variant effect prediction, ChromBPNet reaches AUROC 0.772 (African caQTL) and 0.892 (Yoruban dsQTL), outperforming all DNALM-based methods.
- In zero-shot motif sensitivity, likelihood scoring prioritizes motifs for all DNALMs while embedding cosine distance is near chance (most accuracies 40-60%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DNABERT-2 with LoRA rather than use the ab initio probing-head-like model | Regulatory element identification · Regulatory element identification | paired accuracy: 0.932 -> 0.973 | Helped |
| Use the ab initio ChromBPNet-like classifier rather than probed DNALM embeddings | Cell-type-specific element classification · Cell-type-specific element classification | overall accuracy: 0.420 -> 0.667 | Helped |
| Fine-tune DNALMs with LoRA rather than freeze them and train a probe | Cell-type-specific element classification · Cell-type-specific element classification | overall accuracy: 0.420 -> 0.671 | Helped |
| Use fine-tuned Nucleotide Transformer rather than ChromBPNet | Chromatin activity prediction · GM12878 | Spearman r among positives: 0.540 -> 0.515 | Hurt |
| Use fine-tuned Caduceus rather than ChromBPNet | Chromatin activity prediction · H1ESC | Spearman r among positives: 0.754 -> 0.744 | Hurt |
| Use fine-tuned Nucleotide Transformer rather than ChromBPNet | Chromatin activity prediction · K562 | Spearman r among positives: 0.574 -> 0.583 | Helped |
| Fine-tune Nucleotide Transformer rather than use its frozen embedding probe | Chromatin activity prediction · K562 | Spearman r among positives: 0.499 -> 0.583 | Helped |
| Use ChromBPNet rather than fine-tuned Caduceus for counterfactual variant scoring | Variant scoring · African | AUROC: 0.650 -> 0.772 | Helped |
| Use ChromBPNet rather than fine-tuned Nucleotide Transformer for counterfactual variant scoring | Variant scoring · Yoruban | AUROC: 0.670 -> 0.892 | Helped |
| Fine-tune Caduceus rather than probe frozen representations | Variant scoring · African | AUROC: 0.512 -> 0.650 | Helped |
| Use likelihood scoring rather than embedding scoring for zero-shot TF motif sensitivity | DNALMs · TF motif sensitivity | accuracy | Helped |
| Use final-layer embeddings and cosine distance rather than likelihood scoring for TF motif sensitivity | DNALMs · TF motif sensitivity | accuracy | Hurt |
Surprising findings
- Embedding-based zero-shot methods fail to distinguish TF motifs from background, performing near chance for most motifs.
- Simpler ab initio CNNs match or exceed much larger fine-tuned DNALMs on several tasks despite far fewer parameters.
Limitations
- Evaluation limited to short local contexts (up to 2 kb) and does not cover distal regulatory interactions, gene expression, or 3D genome architecture.
- Only human-focused tasks and a limited set of regulatory element types; does not cover UTRs or splicing.
Experimental setup
Datasets: ENCODE cCREs HOCOMOCO v12 ENCODE ATAC-seq peaks ENCODE DNase-seq peaks caQTL African LCL dataset dsQTL Yoruban LCL dataset
Benchmarks: DART-Eval
Metrics: accuracy AUROC AUPRC Spearman r Pearson r adjusted mutual information
Baselines: ChromBPNet Probing-head-like CNN ChromBPNet-like CNN
Builds on: ChromBPNet LoRA HOCOMOCO ENCODE cCREs
Closest papers by meaning
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.89)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.86)
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.86)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.85)
- Absorb & Escape: Overcoming Single Model Limitations in Generating Heterogeneous Genomic Sequences (NeurIPS 2024 · similarity 0.85)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.