Research map / Architectures and efficiency / CNN design
Revisiting Convolution Architecture in the Realm of DNA Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Revisits CNNs for DNA foundation models by proposing ConvNova, a dilated, gated, dual-branch convolutional network that beats recent Transformer and SSM DNA models on over half of the evaluated tasks with fewer parameters.
Problem
DNA foundation models are dominated by Transformer and SSM architectures, but CNNs have not been systematically compared with them on foundation model benchmarks, so it is unclear whether CNNs are surpassed.
Approach
A simple CNN with three design choices: dilated convolutions (instead of downsampling) to enlarge the receptive field, gated convolution blocks, and a dual-branch structure in which one branch supplies gating signals to the other.
Key results
- On the NT benchmark, ConvNova (1.7M params) achieves best results on 12 of 18 tasks and outperforms the second-best method by 10.5 points on H3K4me3 and by an average of 5.8% on histone tasks.
- On BEND gene finding, ConvNova (7.4M) reaches MCC 0.55, versus 0.52 for GENA-LM (336M), 0.43 for DNABERT-2, and 0.35 for HyenaDNA.
- On chromatin profile prediction, pretrained ConvNova (27.4M) reaches average AUC 92.47, above HyenaDNA (6.5M, 91.23) and NTv2 (500M, 92.30 avg reported as 92.30 for 500M is 92.30 vs 92.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConvNova instead of the strongest listed baseline | NT Benchmark · H3K4me3 | MCC: 56.72 -> 67.15 | Helped |
| use ConvNova instead of NTv2 | NT Benchmark · Splice Acceptor | F1: 97.99 -> 96.23 | Hurt |
| use ConvNova instead of the strongest listed baseline | GenomicsBenchmarks · Human Enhancers Cohn | top-1 accuracy: 0.747 -> 0.743 | Hurt |
| use ConvNova instead of GENA-LM | Gene Finding · Gene Finding | MCC: 0.52 -> 0.55 | Helped |
| remove the gate mechanism (ablation) | ConvNova · H3K4me1 (NT Benchmark) | MCC: 56.60 -> 57.83 | Hurt |
| replace the dual-branch design with a single branch (ablation) | ConvNova · H3K4me1 (NT Benchmark) | MCC: 56.60 -> 56.45 | Hurt |
| replace dilation with U-Net-style downsampling (ablation) | ConvNova · Splice Sites All (NT Benchmark) | F1: 96.45 -> 45.65 | Hurt |
| increase receptive field from 15% to 100% (ablation) | ConvNova · H3K4me3 (NT Benchmark) | MCC: 67.15 -> 60.20 | Hurt |
| increase receptive field from 15% to 100% (ablation) | ConvNova · H3K9ac (NT Benchmark) | MCC: 65.49 -> 68.10 | Helped |
| increase dilation rate and kernel size (ablation) | ConvNova · NT Benchmark (18-task average) | average performance: 63.710 -> 74.835 | Mixed |
| modify RoPE theta values and initialize K/Q projection biases to emphasize neighboring nucleotides | NTv2 · H3K4me3 | MCC: 21.40 -> 34.62 | Helped |
| increase ConvNova parameter count (ablation) | ConvNova · Chromatin Profile Prediction (average) | AUC: 91.85 -> 92.47 | Helped |
Surprising findings
- Downsampling (U-Net style) severely degrades CNN performance on DNA tasks, e.g. splice sites all drops by about 50 points, while dilation avoids this.
- For histone tasks H3K4me2, H3K4me3 and H3K14ac, smaller receptive fields (about 15% of input) perform better than global receptive fields.
Limitations
- Does not consider multi-species pretraining data.
- Pretraining restricted to known functional regions was not explored.
- Benchmarks are primarily classification tasks, limiting task diversity.
Experimental setup
Datasets: HG38 Nucleotide Transformer benchmark Genomic Benchmarks BEND gene finding (GENCODE) DeepSEA chromatin profile dataset
Benchmarks: Nucleotide Transformer benchmark Genomic Benchmarks BEND gene finding Chromatin Profile Prediction
Metrics: MCC F1-score top-1 accuracy AUC
Baselines: NTv2 HyenaDNA DNABERT-2 Caduceus-Ph GENA-LM NT-H DeepATT Basenji LegNet CNN (Genomic Benchmarks)
Builds on: HyenaDNA Hyena U-Net gated convolution (partial convolution gating) Caduceus
Closest papers by meaning
- ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopy (ICML 2025 · similarity 0.85)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.85)
- Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
- Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding (NeurIPS 2024 · similarity 0.84)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.84)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.