Research map / Multi-agent systems and games / LLM agents

Protein Design with Agent Rosetta: A Case Study for Specialized Scientific Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Agent Rosetta, an LLM agent operating Rosetta via a structured RosettaScripts environment (templated actions, surrogate metrics, multi-turn refinement), matching ProteinMPNN/BoltzGen on canonical-residue backbone design and beating a human protocol on non-canonical TRF insertion.

Problem

Protein design with Rosetta is powerful for non-canonical residues but hard to use because RosettaScripts XML is unconventional; ML design models are restricted to canonical amino acids and narrow pipelines. Prompting alone fails to get frontier LLMs to produce valid Rosetta actions.

Approach

A single LLM agent iteratively refines designs through a multi-turn loop in which the environment provides typed action templates (rotamer_change, backbone_change, go_back_to), simplified composition-penalty syntax, on-demand documentation, Pareto selection over ensembles, and error feedback, rather than having the LLM write raw XML.

Key results

• With the environment, all frontier LLMs reach >=85% action success; designs are within 0.20 Å RMSD of ProteinMPNN and BoltzGen, with Gemini 2.5 Flash and Qwen3 235B achieving the best cost-performance.
• On TRF insertion, Agent Rosetta with GPT-5 outperforms the human baseline in AF3 RMSD and pLDDT; the human baseline included no TRF on 8UZL while GPT-5 reached ~14.5%.
• Without the environment, prompting alone fails: GPT-5 gets ~70% on original composition-penalty syntax, and frontier models fail the hard ad-hoc RosettaScripts tasks (0% on Task 4).

Surprising findings

• Prompt engineering alone fails to produce semantically correct RosettaScripts composition penalties, even for GPT-5 (~70% best).
• Removing ESMFold metrics from the context made no significant change in performance for Qwen3 235B.

Limitations

• Limited to RosettaScripts and a few representative PDB structures
• Self-correction loops where the agent alternates between wrong syntax or option names
• Qwen3 30B and other smaller models show a notable action success gap; NCAA performance depends on valid ResidueSelector segments

Experimental setup

Datasets: PDB structures (8 canonical-design backbones, 4 de novo NCAA-design folds)

Benchmarks: fixed-backbone sequence design (8 PDB backbones) TRF non-canonical inclusion (4 de novo PDBs) environment ablation (4 verifiable RosettaScripts tasks)

Metrics: ESMFold RMSD ESMFold pLDDT action success rate AF3 RMSD AF3 pLDDT MD secondary structure change TRF inclusion rate cost per run

Baselines: ProteinMPNN BoltzGen one-shot human Rosetta protocol staged human Rosetta protocol deterministic scripted policy randomized scripted policy

Builds on: Rosetta RosettaScripts ProteinMPNN ESMFold GRPO

Closest papers by meaning

• ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design (NeurIPS 2026 · similarity 0.87)
• Searching for High-Value Molecules Using Reinforcement Learning and Transformers (ICLR 2024 · similarity 0.87)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.86)
• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery (ICML 2024 · similarity 0.86)
• Steering Protein Language Models (ICML 2025 · similarity 0.86)
• Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning (NeurIPS 2026 · similarity 0.85)