Research map / Large language models / Agents, code and math

Understanding the Challenges in Iterative Generative Optimization with LLMs

NeurIPS 2026 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies three hidden design choices (starting artifact, credit horizon, experience batching) in LLM-based generative optimization loops via case studies on MLAgentBench, Atari, and BBEH.

Problem

Generative optimization with LLMs is brittle in practice; setting up a learning loop requires hidden design choices about what the optimizer can edit and what learning evidence to provide per update.

Approach

Identifies three learning-loop design factors (starting artifact, credit horizon of execution traces, and batching of trials/errors into learning evidence) and shows through case studies that each materially changes outcomes, linking them to classical ML analogues (initialization, truncated BPTT, minibatch SGD).

Key results

• On Spaceship Titanic, the best one-function pipeline surpassed 72.7% of leaderboard submissions versus 86.6% for the many-function initialization; on Housing Price the ordering flipped (75.6% vs 54.6%).
• On Atari, multi-step credit horizon outperformed one-step in 4 of 8 games (Pong, Breakout, Space Invaders, Asterix) and one-step was better in the other 4.
• On BBEH, the best batch size varied by task (e.g., batch 1 best for Movie Recommendation at 0.889, batch 3 for Geometric Shapes at 0.389, batch 5 for Causal at 0.531); larger batches did not monotonically improve test accuracy.

Surprising findings

• Truncated credit horizons (e.g., 1-step) matched or beat full-episode traces in half of the Atari games.
• Optimizer overfitting (meta-overfitting) appeared in ML pipeline optimization even without numerical gradients.

Limitations

• Only three case studies and one main optimizer (OptoPrime) with one LLM backend (Claude Sonnet-3.5), limiting generality.
• Did not ablate feedback design, choice of optimizer LLM, or optimization procedure.
• Cross-framework check covers only three BBEH tasks with 3 trials and large standard errors.

Experimental setup

Datasets: MLAgentBench Housing Price MLAgentBench Spaceship Titanic Atari (ALE via OCAtari) BigBench Extra Hard

Benchmarks: MLAgentBench Atari BigBench Extra Hard

Metrics: Kaggle leaderboard percentile game score normalized to random and human test accuracy

Baselines: ResearchAgent DQN PPO DSPy LangGraph

Builds on: Trace OptoPrime CleanRL OCAtari MLAgentBench

Closest papers by meaning

• Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs (NeurIPS 2024 · similarity 0.90)
• Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering with Generative Optimization (NeurIPS 2026 · similarity 0.88)
• Position: We Need An Algorithmic Understanding of Generative AI (ICML 2025 · similarity 0.86)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.86)
• Commit0: Library Generation from Scratch (ICLR 2025 · similarity 0.86)
• EvoOptiGraph: Weakness-Driven Coevolution via Graph-Based Structural Generation for Optimization Modeling (NeurIPS 2026 · similarity 0.86)