Research map / Probabilistic and scientific modeling / Molecules and proteins
Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Equi-mRNA, a codon-level mRNA language model that embeds synonymous codons as SO(2) rotations of amino-acid base vectors, with an auxiliary equivariance loss and SO(2)-aware pooling, pretrained on 25M coding sequences.
Problem
Standard language models over mRNA codons do not explicitly encode the symmetry structure of synonymous codons, which affects translation efficiency and gene expression modeling.
Approach
Treat synonymous codon sets as cyclic groups and embed codons as planar rotations (SO(2), extended to Stiefel-basis subspaces and fuzzy angle distributions) of a shared amino-acid base vector, enforced by an auxiliary equivariance loss and symmetry-aware pooling.
Key results
- Fuzzy-theta Stiefel with equivariance variant achieves the highest gains in ablation, e.g. MLOS 0.691 vs vanilla 0.633 and Tc-Riboswitch 0.736 vs 0.698.
- Equi-mRNA (15M, GPT-2) reaches E. coli 0.613, MLOS 0.710, Tc-Riboswitch 0.737 and outperforms HELM and CodonBert on most of six benchmarks with about 30% of HELM's parameters.
- Generative FBD at T=1.0 on iCodon drops to about 580 for the fuzzy variant, a 4.3x gain over vanilla; MSE reduction of up to 28% reported for property retention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use symmetry-aware Equi-mRNA (15M) instead of HELM (Antibody) | Downstream property prediction · iCodon | Spearman correlation: 0.529 -> 0.537 | Helped |
| Use symmetry-aware Equi-mRNA (15M) instead of HELM (Antibody) | Downstream property prediction · SARS-CoV-2 degradation | Spearman correlation: 0.789 -> 0.791 | Helped |
| Use fuzzy θ, Stiefel basis, and equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · E. coli | accuracy: 0.580 -> 0.605 | Helped |
| Use fuzzy θ, Stiefel basis, and equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · MLOS | Spearman correlation: 0.633 ± 0.14 -> 0.691 ± 0.14 | Helped |
| Use fuzzy θ, Stiefel basis, and equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · Tc-Riboswitch | Spearman correlation: 0.698 -> 0.736 | Helped |
| Use fuzzy θ, Stiefel basis, and equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · mRFP | Spearman correlation: 0.797 -> 0.844 | Helped |
| Use fuzzy θ, Stiefel basis, and equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · SARS-CoV-2 degradation | Spearman correlation: 0.779 -> 0.820 | Helped |
| Use learned θ with Stiefel basis and no equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · E. coli | accuracy: 0.580 -> 0.633 | Helped |
| Use learned θ with Stiefel basis and no equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · mRFP | Spearman correlation: 0.797 -> 0.871 | Helped |
| Use fixed θ with Stiefel basis and no equivariance loss instead of vanilla codon embeddings (ablation) | GPT2 Transformer · Tc-Riboswitch | Spearman correlation: 0.698 -> 0.688 | Hurt |
| Increase pretraining data from 1M to 25M sequences | Equi-mRNA (5M) sequence generation · iCodon | Fréchet BioDistance: 76.13 | Helped |
Surprising findings
- Equi-mRNA (15M) does not win on Tc-Riboswitch, where the 5M hybrid model scores higher (0.764 vs 0.737).
- The vanilla model's FBD increases with sampling temperature while equivariant variants decrease.
Limitations
- Restricted to protein-coding regions and fixed triplet tokenization, ignoring UTRs and non-coding elements.
- Additional parameters and Riemannian optimization overhead for Stiefel and fuzzy variants may hinder scalability.
- Evaluation limited to a narrow set of benchmarks and mostly one organism-level analysis.
Experimental setup
Datasets: RefSeq 25M protein-coding sequences MLOS mRFP E. coli expression Tc-riboswitch iCodon SARS-CoV-2 degradation GRCh38 human coding transcriptome
Benchmarks: MLOS mRFP E. coli expression Tc-riboswitch iCodon SARS-CoV-2 degradation
Metrics: accuracy Spearman correlation Fréchet BioDistance mean squared error
Baselines: Vanilla GPT-2 codon model RNA-FM RNABERT CodonBert HELM GPT2 (Antibody)
Builds on: GPT-2 CodonBERT Stiefel manifold optimization
Closest papers by meaning
- HELM: Hierarchical Encoding for mRNA Language Modeling (ICLR 2025 · similarity 0.88)
- mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design (AAAI 2025 · similarity 0.84)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.84)
- SWAMamba: A Sliding Window Attention Mamba Framework for Predicting Translation Elongation Rates (AAAI 2025 · similarity 0.84)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.84)
- BEACON: Benchmark for Comprehensive RNA Tasks and Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.