Research map / Architectures and efficiency / Attention and transformers
Context-Aware Regularization with Markovian Integration for Attention-Based Nucleotide Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARMANIA, a LLaMA-style sliding-window-attention genomic transformer pretrained with next-token loss plus a first-order transition-matrix loss aligning predicted and sequence-specific bigram statistics, evaluated on 40 genomic tasks.
Problem
Capturing long-range dependencies in nucleotide sequences with transformers is hard because full self-attention is quadratic and does not explicitly enforce global transition consistency.
Approach
Augments next-token prediction with a transition-matrix (TM) loss that aligns the model's predicted first-order nucleotide bigram distribution with the empirical bigram statistics of each input sequence, combined with sliding-window attention for efficiency.
Key results
- On the Nucleotide Transformer enhancer task, CARMANIA reports up to 34% absolute MCC gain over prior models (per paper's claim).
- On BGC classification, CARMANIA reaches 48.4% accuracy versus 41.2% for HyenaDNA (over 7% gain).
- First-order TM loss improves F1 macro on AMR gene family from 0.728 to 0.733 and drug class from 0.931 to 0.942; TM loss helps in 33 of 40 tasks per the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add first-order TM loss (ablation) | CARMANIA · AMR classification tasks | F1 Macro: 0.873 ± 0.130 -> 0.882 ± 0.131 | Helped |
| Add first-order TM loss (ablation) | CARMANIA · Human Genome–Short | BLEU: 0.73 -> 0.77 | Helped |
| Add first-order TM loss (ablation) | CARMANIA · 40 downstream benchmarks | performance: 33 of 40 tasks | Helped |
| Add first-order TM loss (ablation) | CARMANIA · ENHANCER | MCC: Δ = 0.0% | No clear effect |
| Add first-order TM loss (ablation) | CARMANIA · H4AC | MCC: 40.9% relative improvement | Helped |
| Use second-order rather than first-order TM loss (ablation) | CARMANIA · AMR classification tasks | F1 Macro: 0.733 -> 0.701; 0.975 -> 0.943; 0.942 -> 0.915 | Hurt |
| Use full attention rather than sliding-window attention (ablation) | CARMANIA · Human Genome–Short | BLEU: 0.77 -> 0.71 | Hurt |
| Increase TM-loss weight β from 0 to 1.0 (ablation) | CARMANIA · AMR classification tasks | F1 Macro: 0.873 -> 0.882 | Helped |
| Increase TM-loss weight β from 1.0 to 5.0 (ablation) | CARMANIA · AMR classification tasks | F1 Macro: 0.882 -> 0.836 | Hurt |
| Use a wide rather than deep Transformer (ablation) | CARMANIA · Scorpio-Gene-Taxa Test | accuracy: 63.87 ± 19.16 vs. 71.05 ± 18.07 | Helped |
| Use a wide rather than deep Transformer (ablation) | CARMANIA · AMR | F1 Macro: 85.22 ± 14.66 vs. 88.32 ± 13.12 | Helped |
| Increase training sequence length from 10k to 160k (ablation) | CARMANIA · GRCh38 | F1 Macro: 0.765 ± 0.120 -> 0.800 ± 0.138 | Helped |
Surprising findings
- Second-order TM loss degraded performance relative to both first-order TM and no TM, attributed to sparsity of higher-order transitions.
- Adding TM loss gave no measurable gain on enhancer prediction despite the claimed up-to-34% MCC gain on enhancer tasks in the abstract; gains tracked inter-class bigram separability.
Limitations
- Fixed beta=1 balancing weight may not suit rare motifs or datasets with differing entropy; authors suggest adaptive scaling.
- TM loss memory grows with vocabulary size and batch dimension, limiting scalability to larger-vocabulary domains.
- Gains from TM loss are non-uniform and absent on some tasks (e.g., enhancer prediction), and the BGC comparison truncates sequences to 100k bp.
Experimental setup
Datasets: GRCh38 Basic Genome Dataset Scorpio-Gene-Taxa Genomic Benchmarks Nucleotide Transformer tasks (Dalla-Torre et al.) AMR dataset (MEGARes, CARD) MiBiG BGC dataset
Benchmarks: Genomic Benchmarks Nucleotide Transformer tasks Scorpio-Gene-Taxa AMR classification BGC classification
Metrics: accuracy MCC F1 macro BLEU perplexity Hamming similarity FLOPs
Baselines: HyenaDNA Caduceus-PH Caduceus-PS Nucleotide Transformer DNABERT-2 Enformer MetaBERTa ConvNova CNN Mamba
Builds on: LLaMA FlashAttention-2 RoPE Markov chain view of in-context learning
Closest papers by meaning
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.85)
- GeneZip: Region-Aware Compression for Long Context DNA Modeling (NeurIPS 2026 · similarity 0.85)
- Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing (ICML 2025 · similarity 0.85)
- Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNA (NeurIPS 2024 · similarity 0.85)
- Hyperbolic Genome Embeddings (ICLR 2025 · similarity 0.85)
- Exploring Molecular Pretraining Model at Scale (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.