Research map / Large language models / Retrieval and knowledge
A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether pretrained LLMs help principled Bayesian optimization over molecules by using them as fixed feature extractors with GP/Laplace surrogates and by finetuning LoRA with a Laplace approximation over the PEFT weights.
Problem
Whether LLMs carry useful chemistry knowledge for Bayesian optimization over discrete molecular candidate pools, given that prior LLM-based BO work uses non-Bayesian uncertainty heuristics.
Approach
Treats LLMs as fixed feature extractors for standard BO surrogates (GP, Laplace-approximated NN) and generalizes LoRA-Laplace to any PEFT method by conditioning on pretrained LLM weights, yielding a posterior over finetuned LLM surrogates without uncertainty heuristics.
Key results
- Chemistry-specific T5-Chem features (and even fingerprints) give better GAP than general-purpose LLM features (T5-Base, GPT-2-Medium, Llama-2-7b), which tend to underperform the fingerprint baseline.
- On the subsampled Redoxmer task, BO-LIFT with GPT-4 cost USD 75.81 over 5 seeds, while a small chemistry LLM with a Laplace surrogate performed better and far cheaper.
- Finetuning LoRA with LLA improves BO performance for both general and chemistry LLMs on most problems, except T5-Chem on Photovoltaics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use chemistry-specific pretrained features instead of general-purpose LLM features | fixed-feature BO surrogates · molecular BO tasks | GAP | Helped |
| use general-purpose LLM features instead of fingerprint features | fixed-feature BO surrogates · molecular BO tasks | GAP | Hurt |
| use T5-Chem features instead of MolFormer features | fixed-feature BO surrogates · molecular BO tasks | GAP | Hurt |
| use T5-Chem instead of T5-Base, fingerprints, or LA fingerprints | multiobjective BO · Multi-Redox and Multi-Laser | hypervolume | Helped |
| use MolFormer instead of T5-Base or LA fingerprints early in optimization | fixed-feature BO surrogates · Multi-Redox and Multi-Laser, early optimization | hypervolume | Helped |
| use MolFormer instead of either LLM at later optimization stages | fixed-feature BO surrogates · Multi-Redox and Multi-Laser, later optimization | hypervolume | Hurt |
| use the just-smiles prompt instead of other prompts (ablation) | T5-Chem fixed-feature surrogate · molecular BO tasks | BO performance | Helped |
| vary the prompt template (ablation) | fixed-feature BO surrogates · molecular BO tasks | BO performance | Mixed |
| use IUPAC instead of SMILES representation (ablation) | fixed-feature BO surrogates · molecular BO tasks | BO performance | Mixed |
| replace Thompson sampling with expected improvement (ablation) | fixed-feature and finetuned T5-Chem surrogates · molecular BO tasks | GAP | No clear effect |
| finetune the LLM with LoRA and apply the Laplace approximation (ablation) | T5-Base and T5-Chem BO surrogates · molecular BO tasks | GAP | Helped |
| finetune T5-Chem with LoRA (ablation) | T5-Chem BO surrogate · Photovoltaics | BO performance | Hurt |
Surprising findings
- General-purpose LLM features often underperform simple Morgan fingerprints for molecular BO.
- Finetuning the surrogate's cost is dominated by forward passes over the candidate pool rather than training.
Limitations
- Restricted to discrete BO over a predetermined molecule pool; continuous BO left to future work.
- Only chemistry domain studied; the finetuning hyperparameters are shared across all problems, possibly hurting some tasks.
- Optimizing the pretrained weights W via marginal likelihood is left out and would make PEFT training as expensive as full finetuning.
Experimental setup
Datasets: Redoxmer Solvation Laser Photovoltaics Kinase Photoswitches Multi-Redox Multi-Laser
Benchmarks: Redoxmer Solvation Laser Photovoltaics Kinase Photoswitches Multi-Redox Multi-Laser
Metrics: GAP metric best objective value over BO iterations hypervolume wall-clock time per BO iteration
Baselines: Morgan fingerprints GP MolFormer BO-LIFT Llama-2-7b GPT-2-Medium T5-Base
Builds on: linearized Laplace approximation LoRA BO-LIFT Gauche Tanimoto kernel BoTorch
Closest papers by meaning
- De Novo Molecular and Crystal Design with Latent Space Bayesian Optimization (AAAI 2025 · similarity 0.88)
- Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation (NeurIPS 2024 · similarity 0.87)
- Efficient Evolutionary Search Over Chemical Space with Large Language Models (ICLR 2025 · similarity 0.87)
- MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search (NeurIPS 2025 · similarity 0.86)
- MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses (ICLR 2025 · similarity 0.86)
- Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.