Research map / Architectures and efficiency / RNNs and normalization
Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically tests 370 transfer-learning experiments with ESM and CARP protein language models to determine whether downstream gains come from feature reuse that scales with pretraining, or from inductive biases, weight statistics, or low-level features.
Problem
It is unclear how features learned by masked-language-model pretraining of protein language models relate to, and are useful for, downstream protein property and structure prediction tasks, and whether performance scales with PLM size, depth, or pretraining.
Approach
Decomposes transfer-learning gains into feature reuse versus alternative explanations (one-hot baseline, random init, weight-statistics-matched init) and tests scaling along model size, layer depth, and pretraining checkpoint, defining pretrain-downstream alignment as improvement that scales with pretraining.
Key results
- Secondary structure prediction is the only task where pretraining improves performance and aligns with MLM pretraining, with gains scaling with model size, depth, and checkpoint.
- Thermostability, GB1 low-vs-high, and AAV tasks benefit from transfer over baselines but do not scale with depth or pretraining loss; thermostability and AAV two-vs-many scale with size, but randomly initialized models scale similarly.
- Subcellular localization shows no benefit of pretraining over random init, and GB1 sampled and two-vs-rest fail to beat one-hot by 10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer pretrained PLM representations instead of one-hot sequence representations | ESM and CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| transfer pretrained PLM representations instead of randomly initialized representations | ESM and CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| transfer pretrained PLM representations instead of weight-statistics-matched random representations | ESM and CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| transfer deeper PLM layers (ablation) | ESM and CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| use larger pretrained models (ablation) | ESM and CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| use later pretraining checkpoints (ablation) | CARP · SS3 (CB513, TS115, CASP12) | accuracy | Helped |
| transfer pretrained PLM representations instead of baseline representations | ESM and CARP · Thermostability | Spearman's rank correlation | Helped |
| transfer deeper PLM layers (ablation) | ESM and CARP · Thermostability | Spearman's rank correlation | No clear effect |
| use later pretraining checkpoints (ablation) | CARP · Thermostability | Spearman's rank correlation | No clear effect |
| increase model size (ablation) | pretrained and randomly initialized ESM and CARP · Thermostability | Spearman's rank correlation | Helped |
| transfer pretrained PLM representations instead of baseline representations | ESM and CARP · GB1 (low vs high) | Spearman's rank correlation | Helped |
| replace pretrained weights with randomly initialized weights (ablation) | ESM and CARP · GB1 (sampled, low vs high, two vs rest) | Spearman's rank correlation | Mixed |
Surprising findings
- Randomly initialized models scale with model size on thermostability and AAV two-vs-many in similar proportions to pretrained models, so size scaling alone does not indicate feature reuse.
- Stat transfer decreases performance on GB1 tasks relative to one-hot and random init baselines.
Limitations
- Only linear probes on mean-pooled representations were tested due to compute cost; end-to-end finetuning was not evaluated.
- Pretraining dataset effects and alternative pretraining objectives could not be tested due to lack of public models.
- Checkpoints publicly available only for CARP, so checkpoint scaling analysis was not run for ESM.
Experimental setup
Datasets: TAPE secondary structure (CB513, TS115, CASP12) thermostability dataset (48,000 proteins) subcellular localization (DeepLoc) FLIP GB1 FLIP AAV UniRef50
Benchmarks: TAPE secondary structure FLIP GB1 FLIP AAV thermostability subcellular localization
Metrics: accuracy Spearman's rank correlation
Baselines: one-hot baseline random initialization stat transfer
Builds on: BERT masked language modeling ESM CARP FLIP
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.91)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.90)
- Exploring Molecular Pretraining Model at Scale (NeurIPS 2024 · similarity 0.88)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.87)
- Steering Protein Language Models (ICML 2025 · similarity 0.87)
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.