Research map / Large language models / Agents, code and math
MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces fine-grained scientific hypothesis discovery as combinatorial search and proposes Hierarchical Heuristic Search (HHS, also called MOOSE-Chem2) that uses LLM pairwise comparisons as a reward signal over a chemistry hierarchy, evaluated on a post-2024 expert-annotated chemistry benchmark.
Problem
Existing LLM hypothesis generation yields coarse-grained hypotheses lacking methodological and experimental detail; the paper asks how to search for the fine-grained hypothesis an LLM itself judges most promising.
Approach
Frames fine-grained hypothesis generation as hierarchical combinatorial search: greedy edits are proposed level by level from abstract to concrete chemistry details, with LLM pairwise comparisons as the gradient signal, local optima recombined per level, and a claimed smoothing of the reward landscape by hierarchy.
Key results
- HHS-3 reaches 40.35% soft recall and 23.04% hard recall versus 16.60% and 9.92% for greedy search.
- In LLM-judged pairwise comparison, HHS wins 73.53% overall vs greedy search and 53.43% vs greedy with self-consistency (expert: 76.47% and 74.51%).
- A GPT-4o-mini committee beats a mixed committee, which beats a Gemini-1.5-flash committee, under both evaluators.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical search instead of flat greedy search (ablation) | HHS · Chemistry benchmark | overall win rate (LLM): 73.53% | Helped |
| Add recombination/self-consistency to greedy search (ablation) | Greedy Search · Chemistry benchmark | overall win rate (LLM): 54.90% | Helped |
| Add hierarchical decomposition to greedy search with self-consistency (ablation) | Greedy Search + Self-consistency · Chemistry benchmark | overall win rate (LLM): 53.43% | Helped |
| Add in-context RL to HHS-3 (ablation) | HHS-3 · Chemistry benchmark | Soft Recall: 40.35% -> 33.63% | Hurt |
| Add in-context RL to HHS-3 (ablation) | HHS-3 · Chemistry benchmark | Hard Recall: 23.04% -> 21.29% | Hurt |
| Use an ensemble of three identical gradient LLMs instead of one (ablation) | HHS · Chemistry benchmark | Soft Recall: 32.40% -> 40.35% | Helped |
| Use an ensemble of three identical gradient LLMs instead of one (ablation) | HHS · Chemistry benchmark | Hard Recall: 19.95% -> 23.04% | Helped |
| Use an ensemble of three identical gradient LLMs instead of one (ablation) | HHS · Chemistry benchmark | novelty: 25.49% win, 45.59% lose | Helped |
| Use an ensemble of three identical gradient LLMs instead of one (ablation) | HHS · Chemistry benchmark | feasibility: 41.67% win, 29.41% lose | Hurt |
| Use an ensemble of three identical gradient LLMs instead of one (ablation) | HHS · Chemistry benchmark | overall quality: 8.82% win, 82.35% tie | No clear effect |
| Use three GPT-4o-mini gradient instances instead of a mixed-model committee | HHS committee gradient · Chemistry benchmark | overall preference: 29.17% win vs mixed committee; mixed committee 8.33% win vs GPT-4o-mini committee | Helped |
| Use a mixed-model committee instead of three Gemini-1.5-flash gradient instances | HHS committee gradient · Chemistry benchmark | overall preference: 27.08% win vs Gemini-1.5-flash committee | Helped |
Surprising findings
- A single LLM judge (HHS-1) is roughly comparable to a three-instance ensemble on overall quality, with the ensemble mainly improving novelty.
- Mixing diverse LLMs of similar capacity did not beat repeated instances of the strongest model as the judge.
Limitations
- Evaluation limited to chemistry with a 51-paper benchmark.
- Recall metrics are LLM-judged rather than established metrics.
- Hierarchy is manually designed by domain experts and must be re-instantiated per discipline.
Experimental setup
Datasets: TOMATO-Chem
Benchmarks: fine-grained hypothesis benchmark (51 chemistry papers after Jan 2024)
Metrics: pairwise win/tie/lose rate soft recall hard recall reasoning steps
Baselines: ChemCrow SciMON MOOSE MOOSE-Chem Greedy Search Greedy Search + Self-consistency
Builds on: MOOSE-Chem TOMATO-Chem evolutionary search
Closest papers by meaning
- MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses (ICLR 2025 · similarity 0.89)
- LLM-SR: Scientific Equation Discovery via Programming with Large Language Models (ICLR 2025 · similarity 0.89)
- LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models (ICML 2025 · similarity 0.88)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.88)
- Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab (NeurIPS 2025 · similarity 0.88)
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.