Research map / Label-efficient and robust learning / Recognition and long-tail

What DNA Foundation Models Learn Beyond Sequence Composition

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks five frozen DNA foundation model embeddings against normalized k-mer frequency baselines across 57 genomic classification datasets, testing incremental utility, ridge-based linear recoverability, and exact 6-mer-preserving shuffles; NTv3 is the only model with a consistent gain over composition.

Problem

Whether frozen DNA foundation model representations provide predictive information beyond simple k-mer composition on genomic classification and gene-expression tasks.

Approach

Treats normalized k-mer frequency vectors as a required compositional control and tests incremental utility by concatenating 6-mer features with each frozen FM embedding under matched Random Forest protocols, complemented by a ridge decomposition into k-mer-predictable and residual components and exact composition-preserving shuffles.

Key results

• NTv3-650M improves 46 of 57 datasets when concatenated with 6-mer features under the T6 protocol (mean ΔMCC +0.072), while HyenaDNA, DNABERT-2 and Evo2 do not improve mean MCC.
• Under fixed k=6, 6-mer features outperform NTv3 on only 13 of 57 datasets, versus 52 of 57 for Evo2.
• Exact 6-mer-preserving shuffles reduce NTv3 MCC by 0.143 with a frozen probe and 0.079 after retraining, larger than for DNABERT-2 (0.013 retrained).

Surprising findings

• Median ridge R^2 from 6-mers is highest for HyenaDNA (0.75) and Caduceus (0.71), yet this linear recoverability does not track downstream predictive utility.
• HyenaDNA is sensitive to order perturbation but gives no mean incremental gain from concatenation with 6-mers, so order sensitivity is not sufficient for utility.

Limitations

• Frozen last-layer representations only; no fine-tuning or parameter-efficient adaptation evaluated, so conclusions do not bound FM capacity.
• Datasets inherited from a single benchmark with no locus- or chromosome-disjoint splits; only two exploratory regression tasks, with the Caduceus Bulk RNA result unaudited.
• Several experiments (original benchmark, E3, E4, E8, T6) use non-identical pipelines, so effect sizes are not directly comparable; the fixed-6-mer baseline was not independently re-fitted on all 57 tasks.

Experimental setup

Datasets: DNA Foundation Models Benchmark (57 classification datasets) Genomics Long-range Benchmark Bulk RNA Expression Genomics Long-range Benchmark CAGE Prediction

Benchmarks: DNA Foundation Models Benchmark (57 classification datasets) Genomics Long-range Benchmark (Bulk RNA Expression, CAGE Prediction; exploratory)

Metrics: MCC AUROC accuracy R^2

Baselines: fixed 6-mer features multi-k k-mer baseline best-k oracle HyenaDNA DNABERT-2 Caduceus-Ph Evo2-1B

Builds on: DNA Foundation Models Benchmark BEND Genomics Long-range Benchmark

Closest papers by meaning

• Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.87)
• BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.86)
• SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model (ICML 2025 · similarity 0.86)
• DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.86)
• C3P: Contrastive promoter-protein pretraining yields representations capturing bacterial gene regulation (NeurIPS 2026 · similarity 0.85)
• Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.85)