Research map / Large language models / Agents, code and math
Flow-of-Options: Diversified and Improved LLM Reasoning by Thinking Through Options
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flow-of-Options (FoO), a DAG of LLM-generated option nodes traversed with max-value beam search and case-based reuse, instantiated as an agentic AutoML framework that outperforms DS-Agent, SELA, Data Interpreter, and Autogen on tabular and ADME-Tox tasks.
Problem
LLMs show intrinsic biases toward commonly seen options (e.g., random forest in Sklearn), limiting solution diversity and the quality of agentic ML pipelines; existing agentic ML systems also rely on curated human-insight repositories or rigid pipeline/tree structures.
Approach
Enumerate k LLM-generated options per task step into a fully connected layered DAG (the LLM generates options but does not build the graph), then traverse it with max-value edge updates, beam search over top-b options per depth, an LLM consistency checker, and case-based reuse of past FoO networks.
Key results
- FoO-based framework achieves average rank 1.44 on 16 DS tasks (vs. 2.33 for Data Interpreter, the best baseline) and 1.47 on TDC ADME-Tox tasks.
- Development cost about $0.62 and deployment about $0.03 per task with GPT-4o; SELA took about 21 minutes on average.
- On the drug-combination task, FoO-designed solution reduces MAE to 11.85 vs. 13.88 for Autogen and 16.61 for zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Flow-of-Options instead of the strongest reported baseline, Data Interpreter | GPT-4o agentic framework · General data science tasks (16 tasks) | Avg. Rank: 2.33 -> 1.44 | Helped |
| Use Flow-of-Options instead of DeepMol | GPT-4o agentic framework · TDC ADME-Tox tasks (17 tasks) | Avg. Rank: 2.35 -> 1.47 | Helped |
| Use Flow-of-Options instead of Autogen | GPT-4o agentic framework · Drug-drug combination (DC) | MAE: 13.88 -> 11.85 | Helped |
| Use Flow-of-Options instead of Autogen | GPT-4o agentic framework · Drug-target interaction (DTI) | Pearson correlation: 0.32 -> 0.42 | Helped |
| Use Flow-of-Options instead of zero-shot CoT | GPT-4o agentic framework · Math problem case study | Success rate: 0/3 -> 3/3 | Helped |
| Halve beam width in later development iterations (ablation) | Flow-of-Options development · FoO development tasks | Solution quality | Helped |
| Add a consistency checker to identify invalid paths (ablation) | Flow-of-Options execution · Tasks from Table 1 | Execution time reduction: 22.8% | Helped |
| Add a planner adapter for a new task (ablation) | Flow-of-Options deployment · Tasks from Table 1 | Execution time reduction: 93.1% | Helped |
Surprising findings
- Data Interpreter almost always used XGB or RandomForest despite prompts listing several models, indicating model-choice bias.
- Zero-shot GPT-4o and o1 showed a strong bias toward random forest in Sklearn.
Limitations
- Assumes an evaluation metric exists for scoring walks; unclear for non-ML tasks like the math case study.
- Performance depends on LLMs good at structured JSON output and code generation; GPT-3.5 had code failures.
- Several hyperparameters (options per step, plan depth, beam width) must be tuned by the user.
Experimental setup
Datasets: Wild-blueberry-yield Media-campaign-cost Enzyme-substrate Ethanol-concentration Airline-reviews Spaceship-titanic ILI Smoker-status Mohs-hardness Weather Software-defects Jigsaw
Benchmarks: DS-Agent 16 tabular tasks TDC ADME-Tox tasks Drug-drug combination Drug-target interaction Chemical bond prediction OpenAI Gym CartPole MNIST image generation (SSIM)
Metrics: average rank MAE RMSE AUROC accuracy SSIM Rouge cost time
Baselines: DS-Agent AutoGluon SELA Data Interpreter Autogen Zero-shot CoT DeepMol
Builds on: Tree-of-Thoughts Chain-of-Thoughts Case-based reasoning DS-Agent Data Interpreter
Closest papers by meaning
- Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples (ICML 2025 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.86)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.86)
- Agentic Multi-Turn Reasoning: A Fairness Approach (NeurIPS 2026 · similarity 0.86)
- DEPO: Dual-Efficiency Preference Optimization for LLM Agents (AAAI 2026 · similarity 0.86)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.