AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Predicting evolutionary rate as a pretraining task improves genome language model representations

Micaela Elisa Consens, Kevin K Yang, James Brian Hall, Ashley Mae Conard, BO WANG, Lorin Crawford, Alan M Moses, Alex Xijie Lu

ICML 2026 · Large language models › Language, safety and interpretability · approach: Text generation and evaluation metrics · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add PhyloP prediction (CEP) to autoregressive sequence pretraining (ablation)ArGamba (66M) · Functional ROIs vs random regionsbalanced accuracy (%): 71.54 -> 76.25Improved
add PhyloP prediction (MEM) to masked LM pretraining (ablation)Bi-Gamba (4M) · Functional ROIs vs random regionsbalanced accuracy (%): 71.39 -> 76.80Improved
add evolutionary-rate prediction to pretraining (ablation)ArGamba and Bi-Gamba · Functional ROIs vs upstream regionsbalanced accuracy (%): 66.99 -> 71.40 (ArGamba); 65.92 -> 72.65 (Bi-Gamba)Improved
add evolutionary-rate prediction to pretraining (ablation)ArGamba · Multiclass functional region classificationbalanced accuracy (%): 53.97 -> 53.23No clear effect
add MEM to masked LM pretraining (ablation)Bi-Gamba · Multiclass functional region classificationbalanced accuracy (%): 34.53 -> 42.33Improved
add evolutionary-rate prediction to pretraining (ablation)ArGamba and Bi-Gamba · ATG codon context classificationbalanced accuracy (%): 44.59 -> 53.50 (ArGamba); 46.94 -> 56.16 (Bi-Gamba)Improved
rank variants by predicted evolutionary rate instead of sequence likelihood (ablation)Bi-Gamba MEM-only vs MLM-only · ClinVar missenseAUROC: 0.476 -> 0.641Improved
rank variants by predicted evolutionary rate instead of sequence likelihood (ablation)ArGamba · gnomAD rare vs common missenseAUROC: 0.454 -> 0.589Improved
add NTP+CEP joint training vs CEP-only (ablation)ArGamba · VEP non-coding benchmarksAUPRC: 0.158 (CEP-only) vs 0.156 (NTP+CEP) on OMIMNo clear effect
add MEM to MLM training, scoring by sequence likelihood (ablation)Bi-Gamba · COSMIC missenseAUPRC: 0.022 -> 0.013Hurt
add evolutionary rate-only pretraining vs sequence-only (ablation)ArGamba · Multiclass functional region classificationbalanced accuracy (%): 53.97 -> 50.31Hurt
use alternative loss instead of Gaussian NLL (ablation)Gamba · downstream evaluationperformanceNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy