Research map / Large language models / Language, safety and interpretability
HELM: Hierarchical Encoding for mRNA Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HELM, a pre-training strategy that reweights masked/causal language modeling loss by a codon synonymity hierarchy (hierarchical cross-entropy) for mRNA sequences, improving property prediction, generation and antibody region annotation.
Problem
Standard natural-language LM pre-training ignores the hierarchical codon structure of mRNA (codons grouped by amino acid and synonymy), limiting performance on mRNA property prediction and generation.
Approach
Applies hierarchical cross-entropy over a codon-to-amino-acid tree, weighting prediction errors by hierarchy height (lambda = exp(-alpha h)), so confusing synonymous codons is penalized less than confusing codons for different amino acids; requires no architecture or tokenization change.
Key results
- HELM Transformer MLM improves over XE MLM on all six property prediction datasets (e.g., MLOS 0.653 to 0.701, Tc-Riboswitches 0.569 to 0.626).
- HELM CLM beats XE CLM on five of six property datasets; authors report ~8% average improvement over baselines.
- HELM yields better FBD than XE across temperatures and 2% to 31% MSE reduction in preserving functional properties of generated sequences.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add codon-hierarchy-aware HXE loss to Transformer MLM (ablation) | Transformer mRNA LM · mRNA property prediction, six datasets | Spearman rank correlation: 0.748 -> 0.767 (Ab1); 0.599 -> 0.609 (Ab2); 0.653 -> 0.701 (MLOS); 0.503 -> 0.525 (iCodon); 0.569 -> 0.626 (Tc-Riboswitc | Helped |
| add codon-hierarchy-aware HXE loss to Transformer CLM (ablation) | Transformer mRNA LM · mRNA property prediction | Spearman rank correlation: 0.752 -> 0.760 (Ab1); 0.597 -> 0.614 (Ab2); 0.611 -> 0.592 (MLOS); 0.498 -> 0.529 (iCodon); 0.531 -> 0.619 (Tc-Riboswitc | Mixed |
| add codon-hierarchy-aware HXE loss (ablation) | Ab-mRNA region annotation model · Ab-mRNA region annotation | annotation accuracy: 78.68 -> 83.39 (MLM); 67.51 -> 73.39 (CLM) | Helped |
| add codon-hierarchy-aware HXE loss (ablation) | mRNA representation model · Synonymous sequence clustering | Silhouette score: 0.74 -> 0.91 (MLM); 0.68 -> 0.83 (CLM) | Helped |
| use codon-level instead of nucleotide-level tokenization (ablation) | 50M GPT-2 MLM · Ab1, Ab2, MLOS, Tc-Riboswitches | Spearman rank correlation: 0.699 -> 0.747 (Ab1); 0.569 -> 0.599 (Ab2); 0.528 -> 0.654 (MLOS); 0.533 -> 0.570 (Tc-Riboswitches) | Helped |
| use 6-mer instead of codon-level tokenization (ablation) | 50M GPT-2 MLM · mRFP | Spearman rank correlation: 0.754 -> 0.857 | Helped |
| increase HXE α from 0.2 to 0.4 (ablation) | Transformer HELM (MLM) · mRNA property prediction | Spearman rank correlation: 0.767 -> 0.767 (Ab1); 0.609 -> 0.608 (Ab2); 0.525 -> 0.511 (iCodon); 0.822 -> 0.831 (mRFP); 0.626 -> 0.653 (Tc-Riboswitc | Mixed |
| increase HXE α from 0.2 to 0.6 (ablation) | Transformer HELM (MLM) · mRNA property prediction | Spearman rank correlation: 0.767 -> 0.762 (Ab1); 0.609 -> 0.599 (Ab2); 0.525 -> 0.508 (iCodon); 0.822 -> 0.796 (mRFP); 0.626 -> 0.582 (Tc-Riboswitc | Hurt |
| increase HXE α from 0.2 to 0.4 (ablation) | Transformer HELM (CLM) · mRNA property prediction | Spearman rank correlation: 0.760 -> 0.762 (Ab1); 0.614 -> 0.608 (Ab2); 0.529 -> 0.527 (iCodon); 0.849 -> 0.841 (mRFP); 0.619 -> 0.551 (Tc-Riboswitc | Mixed |
| increase HXE α from 0.2 to 0.6 (ablation) | Transformer HELM (CLM) · mRNA property prediction | Spearman rank correlation: 0.760 -> 0.752 (Ab1); 0.614 -> 0.592 (Ab2); 0.529 -> 0.526 (iCodon); 0.849 -> 0.798 (mRFP); 0.619 -> 0.580 (Tc-Riboswitc | Hurt |
| increase model size from 50M to 100M parameters with XE (ablation) | Transformer XE (CLM) · Ab1, Ab2, MLOS | Spearman rank correlation: 0.752 -> 0.750 (Ab1); 0.597 -> 0.596 (Ab2); 0.611 -> 0.608 (MLOS) | No clear effect |
| add hierarchy at 100M parameters (ablation) | Transformer CLM · Ab1, Ab2, MLOS | Spearman rank correlation: 0.750 -> 0.765 (Ab1); 0.596 -> 0.619 (Ab2); 0.608 -> 0.664 (MLOS) | Mixed |
Surprising findings
- Codon tokenization was not uniformly best: 6-mer tokenization outperformed codon-level on mRFP (0.857 vs 0.754) and iCodon in the appendix.
- Transformer models outperformed Mamba and Hyena on most datasets, attributed to short (~1000 nt) sequences.
Limitations
- Hierarchy is learned in Euclidean space, which may not fully capture tree-like structure; hyperbolic geometry suggested as future work.
- Performance degrades for sequences longer than the ~1400-nucleotide pre-training maximum.
- HELM does not uniformly win: CLM underperforms XE on MLOS (0.592 vs 0.611).
Experimental setup
Datasets: OAS (curated antibody mRNA, 15.3M sequences) Ab1 Ab2 MLOS iCodon Tc-Riboswitches mRFP COVID-19 Vaccine
Benchmarks: Ab1 Ab2 MLOS iCodon Tc-Riboswitches mRFP COV-19 Vaccine Ab-mRNA region annotation
Metrics: Spearman correlation Pearson correlation R^2 Silhouette score Frechet Biological Distance MSE annotation accuracy
Baselines: RNA-FM SpliceBERT CodonBERT One-hot encoding Transformer XE (MLM) Transformer XE (CLM) Hyena XE Mamba XE
Builds on: hierarchical cross-entropy GPT-2 Mamba Hyena
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.88)
- Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models (NeurIPS 2025 · similarity 0.88)
- BEACON: Benchmark for Comprehensive RNA Tasks and Language Models (NeurIPS 2024 · similarity 0.87)
- How to make the most of your masked language model for protein engineering (NeurIPS 2026 · similarity 0.86)
- $\texttt{RNAGenScape}$: property-guided, optimized generation of mRNA sequences with manifold Langevin dynamics (NeurIPS 2026 · similarity 0.86)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.