Research map / Architectures and efficiency / Attention and transformers
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BiMamba and MambaDNA (reverse-complement equivariant Mamba) and uses them to build Caduceus, a family of bi-directional, RC-equivariant DNA foundation models pre-trained with masked language modeling on the human genome.
Problem
Genomic sequence models must capture long-range interactions, bi-directional context, and reverse-complement (RC) symmetry of DNA, which prior long-context DNA LMs (e.g., HyenaDNA, Transformer-based models) handle only partially and at high cost.
Approach
Extends the causal Mamba block to bi-directional BiMamba with shared projection weights, and to MambaDNA by applying the block to a sequence and its reverse complement with shared parameters and channel splitting; Caduceus-PS bakes RC equivariance into the model via parameter sharing and embeddings/LM head, while Caduceus-Ph uses post-hoc conjoining with RC augmentation.
Key results
- Caduceus-PS reaches 0.900 on Human Enhancer Ensembl vs 0.849 for HyenaDNA on the Genomics Benchmarks, and Caduceus models beat HyenaDNA on most histone and regulatory tasks.
- Caduceus-Ph beats attention models with orders of magnitude more parameters on 8 of 18 Nucleotide Transformer tasks.
- On the VEP task, Caduceus-PS exceeds Nucleotide Transformer v2 (500M parameters), and beyond 100k bp from TSS Caduceus outperforms Enformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Caduceus-PS instead of HyenaDNA | Genomics benchmark models · Mouse Enhancers | accuracy: 0.780 -> 0.793 | Helped |
| Use Caduceus-Ph instead of Mamba | Genomics benchmark models · Coding vs. Intergenomic | accuracy: 0.904 -> 0.915 | Helped |
| Use Caduceus-Ph instead of HyenaDNA | Genomics benchmark models · Human vs. Worm | accuracy: 0.964 -> 0.973 | Helped |
| Use Caduceus-Ph instead of Mamba | Genomics benchmark models · Human Enhancers Cohn | accuracy: 0.732 -> 0.747 | Helped |
| Use Caduceus-PS instead of HyenaDNA | Genomics benchmark models · Human Enhancer Ensembl | accuracy: 0.849 -> 0.900 | Helped |
| Use Caduceus-PS instead of HyenaDNA | Genomics benchmark models · Human Regulatory | accuracy: 0.869 -> 0.873 | Helped |
| Use Caduceus-Ph instead of HyenaDNA | Genomics benchmark models · Human OCR Ensembl | accuracy: 0.783 -> 0.828 | Helped |
| Use Caduceus-Ph instead of HyenaDNA | Genomics benchmark models · Human NonTATA Promoters | accuracy: 0.944 -> 0.946 | Helped |
| Add RC equivariance (ablation) | Caduceus · Genomics benchmark tasks overall | accuracy | Mixed |
| Use parameter sharing rather than post-hoc conjoining (ablation) | Caduceus · Genomics benchmark tasks overall | accuracy | Mixed |
| Use Caduceus instead of HyenaDNA | Long-range variant effect prediction models · SNPs more than 100k from nearest TSS | AUROC | Helped |
| Use Caduceus-PS instead of Nucleotide Transformer v2 | Long-range variant effect prediction models · Variant effect prediction across TSS distances | AUROC | Helped |
Surprising findings
- Caduceus-Ph, which uses post-hoc conjoining rather than built-in equivariance, is the best overall model on the Genomics Benchmarks tasks.
- HyenaDNA performs better than Caduceus on splice site annotation tasks.
Limitations
- Evaluation focused on human-genome tasks only.
- Downstream comparisons use small matched-size models; many baselines are taken from prior reported results rather than re-run.
- Splice site tasks favor HyenaDNA, indicating mixed gains.
Experimental setup
Datasets: HG38 human reference genome Genomics Benchmarks Nucleotide Transformer benchmark tasks Enformer-derived variant effect prediction dataset (Chromosomes 9 and 10 test)
Benchmarks: Genomics Benchmarks Nucleotide Transformer tasks variant effect prediction on gene expression (Enformer-derived)
Metrics: top-1 accuracy MCC F1 AUROC
Baselines: HyenaDNA Mamba NTP Nucleotide Transformer v2 DNABERT-2 Enformer CNN
Builds on: Mamba HyenaDNA BERT RCPS (reverse complement parameter sharing) post-hoc conjoining
Closest papers by meaning
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.86)
- DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomes (ICLR 2024 · similarity 0.85)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.84)
- Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis (NeurIPS 2025 · similarity 0.84)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.84)
- GeneZip: Region-Aware Compression for Long Context DNA Modeling (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.