Research map / Large language models / Language, safety and interpretability
Reverse Distillation: Disentangling and Scaling Protein Language Model Representations
Darius Catrina, Christian Bepler, Samuel Sledzieski, Rohit Singh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reverse-distill ESM-2 15B on smaller models | ESM-2 15B · ProteinGym 1 mut (28 datasets) | Spearman: 0.899 -> 0.904 | Improved |
| reverse-distill ESM-2 3B | ESM-2 3B · ProteinGym 1 mut (28 datasets) | Spearman: 0.881 -> 0.893 | Improved |
| reverse-distill ESM-2 650M | ESM-2 650M · ProteinGym 2 mut (28 datasets) | Spearman: 0.672 -> 0.688 | Improved |
| reverse-distill ESM-2 3B | ESM-2 3B · ProteinGym >2 mut set, 1 mut (6 datasets) | Spearman: 0.591 -> 0.574 | Hurt |
| reverse-distill ESM-2 15B | ESM-2 15B · R2/R1 | AUPR: 0.368 -> 0.468 | Improved |
| reverse-distill ESM-2 650M | ESM-2 650M · MIB | AUPR: 0.881 -> 0.855 | Hurt |
| reverse-distill ESM-2 650M | ESM-2 650M · LOC | AUPR: 0.709 -> 0.669 | Hurt |
| reverse-distill ESM-2 3B | ESM-2 3B · SSP Q3 | AUPR: 0.791 -> 0.816 | Improved |
| use progressive chain rather than direct 8M->650M distillation (ablation) | rd.650M · ProteinGym ARGR_ECOLI | Spearman: 0.849 -> 0.858 | Improved |
| use progressive chain rather than direct 650M->3B distillation (ablation) | rd.3B · ProteinGym ARGR_ECOLI | Spearman: 0.859 -> 0.873 | Improved |
| use principal component regression with Johnstone threshold instead of OLS (ablation) | rd models · ProteinGym (37 datasets) | Spearman win rate: PCR wins 58.8% (650M), 60.3% (3B), 63.2% (15B) | Improved |
| reverse distillation vs concatenation + PCA (ablation) | rd.15B · ProteinGym (37 datasets) | Spearman win rate: wins 66.7% (650M), 75.8% (3B), 62.1% (15B) | Improved |
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.90)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.89)
- Concept Bottleneck Language Models For Protein Design (ICLR 2025 · similarity 0.89)
- LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling (NeurIPS 2026 · similarity 0.89)
- Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (ICML 2024 · similarity 0.88)
- ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
