Research map / Architectures and efficiency / Attention and transformers
Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the xLSTM architecture into three domain-specific variants (DNA-xLSTM, Prot-xLSTM, Chem-xLSTM) for generative modeling, representation learning, and in-context learning on DNA, protein, and SMILES sequences.
Problem
Transformer-based language models for biological and chemical sequences scale quadratically with sequence length, limiting long genomic contexts and in-context learning on proteins and molecules.
Approach
Tailor xLSTM (sLSTM/mLSTM blocks with linear-time recurrence) to biological sequences: reverse-complement equivariant blocks for DNA, fill-in-the-middle homology-conditioned training with RoPE for proteins, and domain-conditioned in-context learning for SMILES, plus an efficient bidirectional mLSTM variant.
Key results
- DNA-xLSTM-2M-PH outperforms similarly sized HyenaDNA and Mamba models on 12 of 18 Nucleotide Transformer tasks, and is comparable to the 500M-parameter NT-v2.
- Prot-xLSTM-102M reaches a FIM-token perplexity of 5.52 versus 6.47 for ProtMamba-107M, despite using about a quarter of the training tokens.
- Chem-xLSTM achieves the lowest FCD (0.13) for unconditional 15M-parameter SMILES generation versus 0.15 for SMILES-GPT, with perplexity 1.68.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DNA-xLSTM instead of Mamba or HyenaDNA for pre-training with CLM or MLM | DNA sequence modeling · Human reference genome, 1,024-token context | cross-entropy loss | Helped |
| Use DNA-xLSTM-PH instead of Mamba-PH on Nucleotide Transformers classification tasks | 2M-parameter DNA models · Nucleotide Transformers Tasks | MCC, F1, or accuracy | Mixed |
| Use Prot-xLSTM-102M instead of ProtMamba-107M | Homology-aware protein language modeling · Protein homology-conditioned generation — all tokens | perplexity: 7.47^{\pm0.26} -> 6.83^{\pm0.25} | Helped |
| Use Prot-xLSTM-102M instead of ProtMamba-107M | Homology-aware protein language modeling · Protein homology-conditioned generation — first seq tokens | perplexity: 13.04^{\pm0.36} -> 13.36^{\pm0.35} | Hurt |
| Use Prot-xLSTM-102M instead of ProtMamba-107M | Homology-aware protein language modeling · Protein homology-conditioned generation — FIM tokens | perplexity: 6.47^{\pm0.23} -> 5.52^{\pm0.20} | Helped |
| Use Chem-xLSTM instead of SMILES-GPT | Unconditional molecule generation · ChEMBL-derived SMILES | FCD: 0.15^{\pm{<0.01}} -> 0.13^{\pm{<0.01}} | Helped |
| Use Chem-xLSTM instead of SMILES-GPT | Unconditional molecule generation · ChEMBL-derived SMILES | perplexity: 1.65^{\pm{0.6}} -> 1.68^{\pm{1.0}} | Hurt |
| Use reverse-complement parameter sharing (PS) for DNA-xLSTM instead of the non-equivariant model with RC augmentation (ablation) | DNA-xLSTM · DNA pre-training, MLM | cross-entropy loss | Mixed |
| Increase the DNA-xLSTM context size to 32k using mLSTM (ablation) | DNA-xLSTM · DNA pre-training, CLM and MLM | cross-entropy loss | Helped |
| Train Prot-xLSTM-102M with context up to $2^{18}$ instead of $2^{17}$ (ablation) | Prot-xLSTM-102M · Protein homology-conditioned generation — all tokens | perplexity: 6.83^{\pm0.25} -> 6.65^{\pm0.25} | Helped |
Surprising findings
- Chem-xLSTM does not give the best perplexity in unconditional generation (SMILES-GPT has 1.65), even though it has the best FCD.
- Prot-xLSTM-26M underperforms on HMMER score relative to ProtMamba-28M in the homology-conditioned generation KS test.
Limitations
- Hyperparameters were selected manually due to limited compute, so optimal configurations may not be found.
- Character-level DNA tokenization and amino-acid tokenization without structural information may limit performance and scaling.
- Training datasets carry population, sequence, and chemical exploration biases that limit generalization; perplexity is used as a proxy for downstream performance.
Experimental setup
Datasets: human reference genome OpenProteinSet ChEMBL molecular domains dataset Genomic Benchmark Nucleotide Transformer tasks ProteinGym
Benchmarks: Nucleotide Transformer tasks Genomic Benchmark ProteinGym ChEMBL SMILES generation (FCD, perplexity)
Metrics: MCC F1 accuracy cross-entropy loss perplexity Kolmogorov-Smirnov statistic Spearman correlation FCD
Baselines: HyenaDNA Caduceus (Mamba-PS/PH) Transformer++ ProtMamba-28M ProtMamba-107M SMILES-LSTM SMILES-GPT SMILES-S4 Chem-Mamba ESM-2 ProGen2 Nucleotide Transformer
Builds on: xLSTM Mamba Caduceus ProtMamba HyenaDNA
Closest papers by meaning
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.87)
- Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks (ICML 2025 · similarity 0.87)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.87)
- LICO: Large Language Models for In-Context Molecular Optimization (ICLR 2025 · similarity 0.87)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.87)
- Concept Bottleneck Language Models For Protein Design (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.