Research map / Large language models / Agents, code and math

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH

NeurIPS 2026 · Large language models › Agents, code and math · approach: LLMs for optimization and heuristics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LB-MCTS, which uses an MCTS tree as shared state for CASH so that per-algorithm GP surrogates and GPT-4o-mini LLM proposers alternate under a learned BO-selection probability, evaluated on 104 AMLB datasets.

Problem

Combined algorithm selection and hyperparameter optimization (CASH) is high-dimensional and hierarchical, where existing LLM-based and LLM-BO hybrid optimizers generalize poorly and mix cross-algorithm histories.

Approach

Represent CASH optimization as a tree search where each path is a hierarchical trajectory; per-algorithm GP surrogates and LLM proposals share this tree, with Selective Tuning Memory retrieving relevant trials and a Kendall-tau-based policy shifting proposals from LLM to BO as the surrogate improves.

Key results

• LB-MCTS achieves a final average validation rank of 2.16 across 104 AMLB datasets, versus 3.49 for the strongest baseline RB.
• On test sets LB-MCTS attains average rank 3.05 and is statistically significantly superior per the Nemenyi CD diagram.
• Ablation on 10 datasets: LB-MCTS reaches rank 1.5 among directive variants and 1.6 in the BO-LLM ablation versus 2.3 for fixed 50:50 mixing.

Surprising findings

• LLM-based baselines are competitive only in roughly the first 50 iterations and then deteriorate to ranks 5.27-6.63.
• LB-Flat (unstructured history sharing) performs worse (rank 3.4) than the tree-structured variants, showing flat history sharing is insufficient.

Limitations

• Validation feedback can be noisy for small or noisy datasets, which can mislead MCTS and the LLM proposer.
• Performance depends on the capability of the underlying LLM; weaker models give less informative proposals.
• Theoretical guarantees rely on assumptions (e.g., power-law LLM regret) that are modeled rather than proven empirically.

Experimental setup

Datasets: AMLB

Benchmarks: AMLB

Metrics: balanced accuracy MSE average rank

Baselines: SMAC OptDivBO Rising Bandit MOSAIC OPRO LLAMBO BOPRO BORA

Builds on: Auto-WEKA SMAC LLAMBO OPRO Reflexion PUCT

Closest papers by meaning

• Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.89)
• Aligning Tree-Search Policies with Fixed Token Budgets in Test-Time Scaling of LLMs (ICML 2026 · similarity 0.88)
• Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees (NeurIPS 2024 · similarity 0.87)
• Decision Tree Induction Through LLMs via Semantically-Aware Evolution (ICLR 2025 · similarity 0.87)
• SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided Search (NeurIPS 2025 · similarity 0.87)
• AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training (ICML 2024 · similarity 0.86)