Research map / Large language models / Language, safety and interpretability

BEACON: Benchmark for Comprehensive RNA Tasks and Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BEACON, a 13-task RNA benchmark spanning structure, function, and engineering, evaluates CNN/ResNet/LSTM and RNA language models, studies tokenizer and positional-encoding choices, and proposes the BERT-based BEACON-B baseline.

Problem

Existing RNA deep learning and RNA language models are evaluated on disparate individual datasets, so there is no standardized benchmark for fair comparison across RNA understanding tasks.

Approach

Compile 13 RNA tasks from prior work into one benchmark, evaluate supervised baselines and pretrained RNA LMs under identical fine-tuning settings, ablate tokenizer (single nucleotide, BPE, k-mer, non-overlapping) and positional encoding (APE, ALiBi, RoPE), and use the best combination (single nucleotide + ALiBi, BERT backbone) to pretrain a low-cost baseline, BEACON-B.

Key results

• BEACON-B512 achieves 61.20% P@L on contact map prediction and 72.00% R^2 on APA, beating the listed pretrained RNA LMs on several tasks while using only about 3.6 GPU days.
• Pretrained RNA LMs outperform the previous task-specific SOTA on 8 of 13 tasks according to the paper.
• Single nucleotide tokenization achieves the best performance on 11 of 13 tasks in the component ablation, and ALiBi outperforms RoPE and APE in most comparisons.

Measured effects reported in the paper

Surprising findings

• ResNet and LSTM trained from scratch outperform most pretrained RNA language models on several tasks, e.g. LSTM on MRL (83.94 R^2 vs. lower LM scores on many tasks).
• Single-nucleotide tokenization outperforms 6-mer and BPE tokenization, contrary to the common use of k-mer tokenizers in RNA LMs.

Limitations

• Benchmark may not cover all RNA biology tasks; authors note need for additional tasks and datasets.
• Influence of pretraining data and hyperparameters is not systematically explored.
• Evaluation focuses on predictive accuracy and does not assess interpretability, efficiency, or robustness.

Experimental setup

Datasets: bpRNA-1m RNAcontact StructureImpute icSHAPE SpliceAI splice data APARENT APA data Rfam-based ncRNA data MultiRM Optimus MRL data OpenVaccine Angenent-Mari toehold switch data DeepCRISPR RNACentral

Benchmarks: BEACON (13 tasks)

Metrics: F1 Top-L precision R^2 Top-k accuracy ACC AUC MCRMSE Spearman correlation

Baselines: CNN ResNet LSTM RNA-FM RNABERT RNA-MSM SpliceBERT UTR-LM UTRBERT UFold

Builds on: BERT ALiBi RNA-FM UTR-LM FlashAttention

Closest papers by meaning

• HELM: Hierarchical Encoding for mRNA Language Modeling (ICLR 2025 · similarity 0.87)
• $\texttt{RNAGenScape}$: property-guided, optimized generation of mRNA sequences with manifold Langevin dynamics (NeurIPS 2026 · similarity 0.85)
• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
• BAnG: Bidirectional Anchored Generation for Conditional RNA Design (ICML 2025 · similarity 0.85)
• Deciphering RNA Secondary Structure Prediction: A Probabilistic K-Rook Matching Perspective (ICML 2024 · similarity 0.85)
• CoPRA: Bridging Cross-domain Pretrained Sequence Models with Complex Structures for Protein-RNA Binding Affinity Prediction (AAAI 2025 · similarity 0.85)