Research map / Large language models / Agents, code and math

Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks

ICML 2025 · Large language models › Agents, code and math · approach: LLMs for scientific discovery · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Ehrlich functions, a procedurally generated closed-form test suite for constrained biophysical sequence optimization, and LLOME, a bilevel LLM optimization loop with a novel MargE preference loss, which is compared against LaMBO-2 and a genetic algorithm.

Problem

Evaluating and improving LLMs for highly constrained black-box biomolecular sequence optimization, where existing benchmarks are either expensive (wet lab/simulation) or too easy or leak into pretraining data.

Approach

Use Ehrlich functions (closed-form, provably solvable, tunable difficulty) as test beds, and train an LLM in an outer loop on oracle-labeled data while an inner loop generates candidates without oracle access; the MargE loss (FReKL objective with clipped margin reward and importance sampling from the reference policy) is used for training.

Key results

• LLOME-MargE achieves the best minimum regret on medium-difficulty Ehr(32,32)-4-4-4 (f2), while LaMBO-2 performs best on Ehr(32,128)-4-4-4 (f3).
• On easy and difficult Ehrlich functions, all methods perform comparably, either solving quickly or making little progress.
• Off-the-shelf o1 and Gemini 2.0 Flash Thinking with full specification achieve minimum regrets of 0.9375 and 0 with 8 samples, but most outputs were infeasible or near regret 1.

Measured effects reported in the paper

Surprising findings

• DPO-trained LLOME's likelihood for both preferred and dispreferred sequences continuously declines, and its feasibility worsens over iterations, so DPO appears ill-suited for constrained problems.
• Switching from SFT to MargE at an intermediate point gives the worst performance, suggesting conflicting loss landscapes.

Limitations

• LLM likelihoods are miscalibrated with respect to regret, especially for high-likelihood sequences outside the training distribution.
• LaMBO-2 requires hyperparameter tuning per test function, while LLOME is more robust but more expensive computationally.
• Evaluation is restricted to synthetic Ehrlich functions with the spaced-motif epistasis factor set to zero.

Experimental setup

Datasets: Ehrlich functions (procedurally generated)

Benchmarks: Ehrlich functions Ehr(8,128)-8-8-8 Ehr(32,32)-4-4-4 Ehr(32,128)-4-4-4 Ehr(32,128)-8-8-8

Metrics: minimum simple regret average regret feasibility diversity (unique sequences) reward

Baselines: LaMBO-2 genetic algorithm LLOME-SFT LLOME-DPO REINFORCE o1 Gemini 2.0 Flash Thinking

Builds on: direct preference optimization REINFORCE LaMBO-2 generalized variational Bayes

Closest papers by meaning

• LICO: Large Language Models for In-Context Molecular Optimization (ICLR 2025 · similarity 0.89)
• DePLM: Denoising Protein Language Models for Property Optimization (NeurIPS 2024 · similarity 0.89)
• Training Compute-Optimal Protein Language Models (NeurIPS 2024 · similarity 0.88)
• A Variational Perspective on Generative Protein Fitness Optimization (ICML 2025 · similarity 0.87)
• Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (ICLR 2025 · similarity 0.87)
• Controllable Protein Sequence Generation with LLM Preference Optimization (AAAI 2025 · similarity 0.86)