Research map / Label-efficient and robust learning / Self-supervised and contrastive
mRNA2vec: mRNA Embedding with Language Model in the 5'UTR-CDS for mRNA Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes mRNA2vec, a data2vec-style contextual language model pretrained on concatenated 5' UTR and CDS sequences with location-biased hard masking and auxiliary MFE regression and secondary-structure classification tasks, reporting gains on UTR translation efficiency/expression tasks over UTR-LM and competitive CDS results versus CodonBERT.
Problem
Learning general-purpose mRNA sequence embeddings that predict translation efficiency, expression level, stability and protein production for mRNA design, where prior models treat 5' UTR and CDS separately and rely on masked-token prediction.
Approach
Apply data2vec's teacher-student contextual target learning to mRNA over a joint UTR-CDS input with codon (3-nt) tokenization, bias masking toward the 5' UTR tail and CDS head, and add MFE regression and secondary-structure classification as auxiliary pretext losses.
Key results
- On the 5' UTR Muscle-EL task, mRNA2vec reaches 0.80 Spearman versus 0.61 for UTR-LM and 0.78 for RiNALMo.
- Compared with UTR-LM on TE, mRNA2vec reports improvements of 13%, 12%, and 14% on HEK, PC3, and Muscle datasets.
- On mRNA stability, mRNA2vec improves over CodonBERT from 0.34 to 0.53 Spearman; on mRFP expression results are described as comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use mRNA2vec instead of RiNALMo | 5′ UTR prediction · HEK-TE | Spearman rank: 0.62 -> 0.68 | Helped |
| use mRNA2vec instead of RiNALMo | 5′ UTR prediction · PC3-EL | Spearman rank: 0.60 -> 0.70 | Helped |
| use mRNA2vec instead of CodonBERT | CDS mRNA stability prediction · mRNA stability (CDS) | Spearman rank: 0.34 to 0.53 | Helped |
| add MFE and SS auxiliary losses (ablation) | mRNA2vec pretraining · Muscle-TE | Spearman rank: 0.550 -> 0.573 | Helped |
| add MFE and SS auxiliary losses (ablation) | mRNA2vec pretraining · Muscle-EL | Spearman rank: 0.619 -> 0.662 | Helped |
| use MFE auxiliary loss only (ablation) | mRNA2vec pretraining · PC3-EL | Spearman rank: 0.488 -> 0.473 | Hurt |
| use SS auxiliary loss only (ablation) | mRNA2vec pretraining · mRNA-Stability | Spearman rank: 0.465 -> 0.458 | Hurt |
| increase hard-mask α from 0 to 0.8 (ablation) | mRNA2vec pretraining · pretraining loss | loss: 0.349 -> 0.332 | Helped |
| increase hard-mask α from 0 to 0.5 (ablation) | mRNA2vec pretraining · pretraining | SpermanR: -0.11 -> 0.27 | Mixed |
| use the second-last instead of the last hidden state (ablation) | mRNA2vec downstream embedding · HEK | Spearman rank: 0.389 -> 0.515 | Helped |
| use the second-last instead of the last hidden state (ablation) | mRNA2vec downstream embedding · PC3 | Spearman rank: 0.556 -> 0.619 | Helped |
| use the second-last instead of the last hidden state (ablation) | mRNA2vec downstream embedding · Muscle | Spearman rank: 0.560 -> 0.66 | Helped |
Surprising findings
- Adding secondary structure as a 27-category token-level classification helped, contrary to UTR-LM's finding that secondary structure degraded performance.
- Truncating the 5' UTR input to a shorter rightmost window improved TE results, with full sequences worsening them.
Limitations
- Downstream CDS results are only comparable, not superior, on protein production level.
- Hyperparameters beta1 and beta2 were fixed across all datasets rather than tuned per task.
- Downstream evaluation freezes the encoder and trains only a regressor head, and the Muscle dataset has only about 1k sequences.
Experimental setup
Datasets: NCBI mRNA sequences (five species) Human embryonic kidney (HEK) 293T TE/EL data PC3 TE/EL data Human muscle TE/EL data mRNA stability dataset mRFP expression dataset
Benchmarks: HEK-TE PC3-TE Muscle-TE HEK-EL PC3-EL Muscle-EL mRNA stability mRFP expression
Metrics: Spearman rank correlation
Baselines: RNA-FM RNABERT Cao-RF CodonBERT UTR-LM RiNALMo TF-IDF T5
Builds on: data2vec T5 UTR-LM InfoNCE
Closest papers by meaning
- HELM: Hierarchical Encoding for mRNA Language Modeling (ICLR 2025 · similarity 0.86)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
- Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment (NeurIPS 2025 · similarity 0.85)
- Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models (NeurIPS 2025 · similarity 0.84)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.84)
- Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.