Research map / Vision-language and multimodal / Human and embodied modeling
Multi-modal Transfer Learning between Biological Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IsoFormer, a multi-modal model that fuses pre-trained DNA (Enformer or NT), RNA (NT), and protein (ESM2) encoders via cross-attention to predict transcript isoform expression across GTEx tissues.
Problem
Existing biological foundation models handle a single sequence modality (DNA, RNA, or protein), and gene-level DNA models like Enformer cannot predict isoform-specific expression, which requires multiple modalities.
Approach
Connect modality-specific pre-trained encoders with a cross-attention aggregation module with residual connections, train them jointly end-to-end, and use a linear expression head to predict transcript expression across tissues from DNA, RNA, and protein sequences.
Key results
- IsoFormer with Enformer DNA encoder reaches R^2 0.53 and Spearman 0.72, versus Enformer alone at R^2 0.21 and Spearman 0.46.
- Adding modalities helps: DNA+RNA+protein gives R^2 0.43 vs 0.13 (DNA only), 0.36 (RNA only), and 0.20 (protein only) with NT encoders.
- Randomly initializing all encoders drops R^2 to 0.10, while random init of only the DNA or RNA encoder gives 0.41 and 0.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DNA, RNA, and protein inputs instead of using DNA only (ablation) | IsoFormer with NT-based encoders · GTEx transcript isoform expression prediction | R²: 0.13 ± 0.02 -> 0.43 ± 0.01 | Helped |
| add RNA and protein inputs to DNA (ablation) | IsoFormer with NT-based encoders · GTEx transcript isoform expression prediction | Spearman: 0.43 ± 0.01 -> 0.65 ± 0.01 | Helped |
| add protein input to DNA + RNA (ablation) | IsoFormer with NT-based encoders · GTEx transcript isoform expression prediction | R²: 0.39 ± 0.01 -> 0.43 ± 0.01 | Helped |
| add protein input to DNA + RNA (ablation) | IsoFormer with NT-based encoders · GTEx transcript isoform expression prediction | Spearman: 0.64 ± 0.01 -> 0.65 ± 0.01 | Helped |
| use DNA + RNA instead of DNA + protein (ablation) | IsoFormer with NT-based encoders · GTEx transcript isoform expression prediction | R²: 0.28 ± 0.01 -> 0.39 ± 0.01 | Helped |
| replace NT with Enformer as the DNA encoder (ablation) | IsoFormer · GTEx transcript isoform expression prediction | R²: 0.43 ± 0.01 -> 0.53 ± 0.01 | Helped |
| replace NT with Enformer as the DNA encoder (ablation) | IsoFormer · GTEx transcript isoform expression prediction | Spearman: 0.65 ± 0.01 -> 0.72 ± 0.00 | Helped |
| add Perceiver Resampler before cross-attention (ablation) | IsoFormer aggregation · GTEx transcript isoform expression prediction | R²: 0.53 ± 0.01 -> 0.49 ± 0.04 | Hurt |
| add Perceiver Resampler before cross-attention (ablation) | IsoFormer aggregation · GTEx transcript isoform expression prediction | Spearman: 0.72 ± 0.00 -> 0.69 ± 0.02 | Hurt |
| replace cross-attention with linear projection and Perceiver Resampler (ablation) | IsoFormer aggregation · GTEx transcript isoform expression prediction | R²: 0.53 ± 0.01 -> 0.49 ± 0.02 | Hurt |
| replace cross-attention with C-Abstractor (ablation) | IsoFormer aggregation · GTEx transcript isoform expression prediction | R²: 0.53 ± 0.01 -> 0.53 ± 0.01 | No clear effect |
| remove pre-training from all encoders (ablation) | IsoFormer · GTEx transcript isoform expression prediction | R²: 0.10 -> 0.53 | Hurt |
Surprising findings
- Using Enformer, a gene-expression model, as the DNA encoder outperforms NT, which the authors attribute to better alignment with the expression prediction task.
- The perceiver resampler step, despite being a common multi-modal design, performs worse than plain cross-attention even after hyperparameter search.
Limitations
- Expression targets are averaged GTEx values from a single reference genome sequence, without individual-level variation.
- RNA encoder is the NT model reused with T substituted for U, since no RNA foundation model was publicly available.
- Evaluated only on one task (isoform expression) and one dataset (GTEx).
Experimental setup
Datasets: GTEx Transcript TPMs (v8) Ensembl GRCh38
Benchmarks: GTEx transcript isoform expression (held-out test set split by gene)
Metrics: R^2 Spearman correlation
Baselines: Enformer Nucleotide Transformer ESM2 Perceiver Resampler + Cross-Attn Linear Projection + Perceiver Resampler C-Abstractor
Builds on: Enformer Nucleotide Transformer ESM2 Perceiver Resampler C-Abstractor
Closest papers by meaning
- A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features (ICML 2025 · similarity 0.86)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.85)
- JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model (NeurIPS 2025 · similarity 0.85)
- What DNA Foundation Models Learn Beyond Sequence Composition (NeurIPS 2026 · similarity 0.85)
- BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR 2024 · similarity 0.84)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.