Research map / Large language models / Language, safety and interpretability
Concept Bottleneck Language Models For Protein Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CB-pLM, a masked protein language model with a concept bottleneck layer of 718 interpretable concepts that enables controllable generation, interpretation via a linear decoder, and debugging.
Problem
Protein language models are opaque, making it hard to control generated sequences, interpret what they learned, or debug them, which limits trust in high-stakes drug discovery settings.
Approach
Insert a concept bottleneck layer (one interpretable neuron per concept, with an orthogonal unknown embedding) into a masked language model, feed a linear decoder from concepts and unknown embedding, and intervene on concept values with gradient-based token attribution for coordinate selection.
Key results
- CB-pLM achieves about 3x larger change in desired concept values than other conditional models and over 15% higher control accuracy in single-concept interventions.
- CB-pLM reaches 94% control accuracy on multi-concept (GRAVY and charge) interventions, outperforming the second-best conditional model by 34%.
- CB-pLM 24M/150M/650M/3B perplexity on Mason dataset is 4.29/3.20/2.48/2.50, comparable to conditional baselines and ESM2 (4.84/3.41/3.01 at 150M/650M/3B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CB-pLM instead of other conditional language models for single-concept intervention | Conditional protein language models · 14 biophysical concepts, single-concept intervention | control accuracy: over 15% | Helped |
| Use CB-pLM instead of other conditional language models for single-concept intervention | Conditional protein language models · 14 biophysical concepts, single-concept intervention | change in desired concept value: 3× | Helped |
| Use CB-pLM for multi-concept intervention | Conditional protein language models · GRAVY and charge at pH 7, multi-concept intervention | control accuracy: 94% | Helped |
| Use CB-pLM rather than the second-best conditional model for multi-concept intervention | Conditional protein language models · GRAVY and charge at pH 7, multi-concept intervention | control accuracy: 34% | Helped |
| Use single-step integrated gradients rather than random masking for coordinate selection | CB-pLM · Protein concept interventions | control effectiveness: 2 × | Helped |
| Increase CB-pLM model size (ablation) | CB-pLM · Protein concept interventions | control | No clear effect |
| Change aromaticity normalization to make the concept difficult to learn (ablation) | CB-pLM 24M · Aromaticity intervention | control | Hurt |
| Use CB-pLM instead of other conditional language models | Conditional protein language models · Mason dataset | perplexity | Mixed |
| Use CB-pLM rather than LaMBO-2 for Siltuximab redesign | Protein design models · Siltuximab GRAVY reduction | GRAVY distribution shift | Hurt |
| Use CB-pLM rather than ESM2 or Hydrophilic Resample for Siltuximab redesign | Protein design models · Siltuximab redesign | naturalness | Helped |
Surprising findings
- On Siltuximab GRAVY redesign, CB-pLM ranks second to LaMBO-2 in concept distribution shift, though it is a general-purpose model trained on 718 concepts.
- Increasing model size from 150M to larger CB-pLM improved perplexity but gave no control improvement.
Limitations
- Requires training data annotated with predefined concepts.
- Unknown embedding is needed; reliance on it can let the model ignore concepts and orthogonality only encourages disentanglement.
- Generalization to unseen concept combinations is only shown on a toy color-MNIST task, not on proteins.
Experimental setup
Datasets: UniRef50 SWISS-PROT Mason dataset OAS
Benchmarks: Mason dataset perplexity Siltuximab GRAVY redesign
Metrics: perplexity concept control accuracy change in concept value naturalness (TAP scores) percentage passing naturalness filter
Baselines: C-pLM CC-pLM ESM2 LBSTER LaMBO-2 PropEn Discrete Walk-Jump Sampling Hydrophilic Resample
Builds on: concept bottleneck models concept bottleneck generative models ESM2 integrated gradients
Closest papers by meaning
- Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.90)
- How to make the most of your masked language model for protein engineering (NeurIPS 2026 · similarity 0.89)
- Steering Protein Language Models (ICML 2025 · similarity 0.89)
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.88)
- From Mechanistic Interpretability to Mechanistic Biology: Training, Evaluating, and Interpreting Sparse Autoencoders on Protein Language Models (ICML 2025 · similarity 0.88)
- Elucidating the Design Space of Multimodal Protein Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.