Research map / Large language models / Language, safety and interpretability

How to make the most of your masked language model for protein engineering

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes sequence-centric stochastic beam search for masked protein language models using an approximate pseudo-log-likelihood over all one-edit neighbors, and benchmarks samplers and MLMs in silico and in vitro on antibody therapeutic campaigns.

Problem

Sampling from masked protein language models to optimize antibody properties is under-studied; existing mutation-centric samplers (Gibbs, denoising) are costly and tend to yield unlikely or dysfunctional sequences, and incorporating non-differentiable scoring functions is hard.

Approach

Compute one exact PLL per template and approximate the PLLs of all single-substitution neighbors for free via a masked-marginal approximation, then run stochastic beam search with Gumbel noise over full sequences, with multi-objective smooth Tchebycheff guidance treating models and scorers as black boxes.

Key results

• On Trastuzumab CDR-H3 with ESM2-650M at six edits, unguided beam search reached mean p(binder) ~0.95 versus ~0.43 for denoising and ~0.44 for Gibbs.
• In vitro, STS multi-objective guidance with AbLang2 reached a 100% success rate on the FAb campaign (289 samples, 14 methods), with ESM2-650M and AbLang2 having the best unsupervised success rates.
• Approximate PLL correlated strongly with exact PLL on Trastuzumab one-edit variants (Spearman 0.940 for ESM2-650M, 0.869 for AbLang2).

Surprising findings

• AbLang2 likelihoods were negatively correlated with p(binder) on the Trastuzumab task, requiring supervised guidance.
• AbLang2 produced less humanness-typical antibodies despite being trained on human sequences, while ESM2-650M, not trained on antibodies, had high humanness.

Limitations

• Approximate PLL errs for AbLang2 on germline UCA seeds with framework mutations due to germline memorization.
• Guidance with a single humanness or binding model can reduce humanness and diversity.
• Gibbs and beam search differences are shown mainly on limited seeds and a single campaign.

Experimental setup

Datasets: her2-aff-large (367k) Mason CDR-H3 dataset (34k) OAS antibody sequences in-house scFv therapeutic campaign fresh FAb in vitro campaign (729 samples)

Benchmarks: Trastuzumab CDR-H3 in silico optimization scFv in silico benchmark FAb in vitro antibody campaign

Metrics: predicted p(binder) OASis humanness percentile predicted synthesizability predicted thermostability inter- and intra-seed diversity in vitro success rate (synthesizability and binding) approximate-vs-true PLL Spearman correlation

Baselines: Gibbs sampling denoising sampling Gibbs-argmax Gibbs with PoE guidance pIgGen DiffAb+ SAbDabMLM Sapiens

Builds on: stochastic beam search Kool et al. Gumbel top-k beam search masked-marginal zero-shot scoring Gibbs sampling for protein MLMs ESM-3 denoising sampling

Closest papers by meaning

• Concept Bottleneck Language Models For Protein Design (ICLR 2025 · similarity 0.89)
• Steering Protein Language Models (ICML 2025 · similarity 0.89)
• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.88)
• DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.87)
• Synergy of GFlowNet and Protein Language Model Makes a Diverse Antibody Designer (AAAI 2025 · similarity 0.87)
• Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.86)