Research map / Architectures and efficiency / Attention and transformers
JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JanusDNA, a bidirectional DNA foundation model combining a Mamba-Attention-MoE hybrid with a Janus pretraining objective that predicts every token from its bidirectional context, enabling 1M-bp single-nucleotide-resolution modeling.
Problem
Genomic language models struggle to jointly capture long-range dependencies, bidirectional context, and training efficiency; masked LMs use only ~15% of tokens per loss and autoregressive models are unidirectional, while k-mer tokenization loses single-nucleotide resolution.
Approach
Two independent unidirectional Mamba-FFN/MoE stacks (forward and backward) are fused by a FlexAttention layer with a custom mask so every token is predicted from full left and right context, giving MLM-style bidirectionality with autoregressive-style loss on all tokens.
Key results
- On the Nucleotide Transformer benchmark, JanusDNA (mlp) achieves state-of-the-art on 12 of 18 tasks, surpassing models with up to 250x more activated parameters.
- On DNALONGBENCH eQTL (450k-bp sequences), JanusDNA achieves best AUROC on 8 of 9 tissues, e.g. Muscle Skeletal 0.864 vs Enformer 0.621.
- On the Genomic Benchmark, JanusDNA is best on 3 of 8 tasks (e.g. Human OCR Ensembl 0.828 for Caduceus-Ph vs JanusDNA 0.822 close), with other tasks near the best model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JanusDNA MLP w/o Mid-Attn instead of the strongest listed baseline | eQTL prediction · DNALONGBENCH (9 tissues) | AUROC | Helped |
| Add mid-attention to the Mamba and FFN architecture (ablation) | JanusDNA hybrid architecture · Pretraining perplexity | perplexity | Hurt |
| Add MoE layers in place of some FFN blocks (ablation) | JanusDNA hybrid architecture · Pretraining perplexity | perplexity | Helped |
| Combine mid-attention with MoE layers (ablation) | JanusDNA hybrid architecture · Pretraining perplexity | perplexity | Helped |
| Add mid-attention (ablation) | JanusDNA with MoE · Nucleotide Transformer benchmark | performance | Mixed |
| Use reverse-complement representations at prediction (ablation) | JanusDNA · Nucleotide Transformer histone, enhancer, and promoter tasks | score | Helped |
| Use reverse-complement representations at prediction (ablation) | JanusDNA · Nucleotide Transformer splice-site tasks | score | Hurt |
| Add an MLP after feature-fusion attention (ablation) | JanusDNA · Nucleotide Transformer benchmark | performance | Helped |
| Use Janus rather than causal modeling (ablation) | matched-size DNA sequence model · Middle-token prediction | accuracy | Helped |
| Use Janus rather than causal modeling (ablation) | matched-size DNA sequence model · Last-token prediction | accuracy | Helped |
Surprising findings
- Janus models slightly outperform causal models even on last-token prediction, where both have the same information.
- Mid-attention benefits diminish as hidden dimension grows; at hidden size 72 models without mid-attention outperform those with it.
Limitations
- Pretraining restricted to the human reference genome, limiting data diversity.
- Underperforms the largest models on promoter and splice site tasks, attributed to data scale and diversity.
- Genomic Benchmark is saturated, reducing ability to show differences.
Experimental setup
Datasets: HG38 Genomic Benchmark Nucleotide Transformer Benchmark DNALONGBENCH eQTL
Benchmarks: Genomic Benchmark Nucleotide Transformer Benchmark DNALONGBENCH eQTL
Metrics: top-1 accuracy MCC F1-score AUROC
Baselines: Caduceus-PH Caduceus-PS HyenaDNA ConvNova Enformer DNABERT-2 Nucleotide Transformer v2 CNN
Builds on: Mamba FlexAttention Caduceus masked language modeling Nucleotide Transformer benchmark setup
Closest papers by meaning
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.87)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.86)
- Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling (ICML 2024 · similarity 0.86)
- DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA (NeurIPS 2024 · similarity 0.86)
- Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis (NeurIPS 2025 · similarity 0.85)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.