Research map / Large language models / Language, safety and interpretability
Steering Protein Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts activation steering (activation addition) to protein language models to steer lysozyme-like sequence generation toward thermostability/solubility and introduces ASPO, a training-free protein optimization method that selects mutation sites via steering-vector relatedness scores.
Problem
PLMs struggle to generate or optimize proteins with precisely specified properties; existing control methods (fine-tuning, keyword tags) are costly or inflexible.
Approach
Compute per-layer steering vectors as mean activation differences between proteins with and without a target property and add them to PLM activations at inference; extend to optimization by selecting low-relatedness tokens as mutation sites and predicting replacements with steering.
Key results
- On ESM2-650M, activation steering raises lysozyme thermostability to 82.20 vs 56.48 for the original model and 63.56 for fine-tuning.
- ESM3+ASPO reaches thermostability fitness 88.42 (medium) and 86.43 (hard), above AdaLead (63.56 / 55.16) and PEX.
- ESM2+ASPO and ESM3+ASPO reach GFP fitness of 3.862 and 3.739 (medium), more than double the best baseline (GWG 1.683).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply activation steering instead of fine-tuning | ProLLaMA · Lysozyme-like thermostability generation | thermostability: 57.24 (8.64) -> 67.68 (12.86) | Helped |
| Apply activation steering instead of fine-tuning | ProLLaMA · Lysozyme-like solubility generation | solubility: 0.241 (0.086) -> 0.276 (0.110) | Helped |
| Apply activation steering instead of fine-tuning | ESM2 · Lysozyme-like thermostability generation | thermostability: 63.56 (14.87) -> 82.20 (12.92) | Helped |
| Apply activation steering instead of fine-tuning | ESM2 · Lysozyme-like solubility generation | solubility: 0.356 (0.213) -> 0.494 (0.241) | Helped |
| Apply activation steering instead of fine-tuning | ESM3 · Lysozyme-like thermostability generation | thermostability: 62.82 (14.72) -> 82.06 (12.06) | Helped |
| Apply activation steering instead of fine-tuning | ESM3 · Lysozyme-like solubility generation | solubility: 0.318 (0.215) -> 0.582 (0.264) | Helped |
| Use ESM3 + ASPO instead of the strongest listed baseline | Lysozyme thermostability optimization · Medium difficulty | Fitness: 68.25 (9.35) -> 88.42 (3.98) | Helped |
| Use ESM3 + ASPO instead of AdaLead | Lysozyme thermostability optimization · Hard difficulty | Fitness: 55.16 (9.29) -> 86.43 (9.02) | Helped |
| Use ESM3 + ASPO instead of AdaLead | Lysozyme solubility optimization · Medium difficulty | Fitness: 0.617(0.247) -> 0.654 (0.273) | Helped |
| Use AdaLead instead of ESM3 + ASPO | Lysozyme solubility optimization · Hard difficulty | Fitness: 0.530 (0.283) -> 0.397 (0.247) | Hurt |
| Use ESM2 + ASPO instead of GWG | GFP brightness optimization · Medium difficulty | Fitness: 1.683 (0.641) -> 3.862 (0.329) | Helped |
| Use ESM2 + ASPO instead of GWG | GFP brightness optimization · Hard difficulty | Fitness: 1.510 (0.545) -> 3.907 (0.247) | Helped |
Surprising findings
- Steering performance on ProLLaMA peaks with only 10 samples and declines with larger sets, unlike ESM2/ESM3.
- Over-steering (alpha > 5) severely degrades solubility performance, with complete collapse at alpha=20, while thermostability is only mildly affected.
Limitations
- Optimization baselines were run without surrogate updates from ground-truth fitness, a modified setting that may not match originals.
- ASPO yields lower diversity than baselines on thermostability and GFP tasks.
- Gains from steering multiple properties jointly are smaller than single-property gains.
Experimental setup
Datasets: UniRef50 Meltome Atlas DeepSol
Benchmarks: lysozyme-like generation (thermostability, solubility) GFP brightness optimization lysozyme-like optimization (thermostability, solubility)
Metrics: fitness diversity novelty Dissim_init Dissim_high
Baselines: Original Model Fine-tuning AdaLead PEX GWG GGS
Builds on: activation addition contrastive activation addition AdaLead PEX ProLLaMA
Closest papers by meaning
- Steering Generative Models with Experimental Data for Protein Fitness Optimization (NeurIPS 2025 · similarity 0.90)
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.90)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.89)
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.89)
- Concept Bottleneck Language Models For Protein Design (ICLR 2025 · similarity 0.89)
- How to make the most of your masked language model for protein engineering (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.