Research map / Large language models / Agents, code and math

FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Synthesizes open-ended coding problems by mutating closed-ended competitive programming problems and filtering with an idea-divergence metric, then trains Qwen3.5 models with GRPO on 200 synthetic problems.

Problem

Open-ended coding problems (no known optimum, continuous quality score) are scarce and expensive to build manually, while closed-ended coding has abundant verified data.

Approach

Mutate closed-ended problem formulations (change goal, restrict outputs, generalize inputs) to remove known optima, then admit candidates via an idea-divergence metric measuring whether independent solver solutions use different algorithmic strategies, estimated first by LLM judge and then by execution-based score-vector distance.

Key results

• FrontierSmith-trained Qwen3.5-9B reaches 10.62 Avg@5 on FrontierCS vs 11.17 for FrontierCS-human-curated training and 5.38 for HardTests training.
• On Qwen3.5-27B, FrontierSmith achieves 19.82 Avg@5 on FrontierCS vs 13.98 (FrontierCS-trained) and 661.64 Avg@5 on ALE-bench vs 543.80.
• Idea divergence (LLM estimate) is about 0.40-0.42 for open-ended sources vs 0.14 for HardTests; removing filters drops FrontierCS Avg@5 from 10.62 to 8.57.

Surprising findings

• Training on 200 synthetic problems from closed-ended seeds matches or exceeds training on 172 human-curated open-ended problems on the 27B model and ALE-bench average.
• Random reward training yields near-base performance, so gains are attributed to task-specific reward signals rather than RL dynamics.

Limitations

• Pipeline limited to self-contained algorithmic problems; excludes repo-level or infrastructure-dependent open-ended tasks.
• RL training limited to 100 steps of single-turn GRPO; agentic RL not explored.
• Only 10% of candidates entering cross-validation yield validated test/verifier pairs; coarse filter has 9% false positive and 19% false negative rates on small samples.

Experimental setup

Datasets: HardTests FrontierCS ALE-bench

Benchmarks: FrontierCS ALE-bench-lite

Metrics: Avg@5 Best@5 Elo-based performance rating idea divergence

Baselines: FrontierCS-trained RL ALE-bench-trained RL HardTests-trained RL Random Reward RL

Builds on: GRPO HardTests FrontierCS ALE-bench veRL

Closest papers by meaning

• Planning in Natural Language Improves LLM Search for Code Generation (ICLR 2025 · similarity 0.88)
• ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests (NeurIPS 2025 · similarity 0.87)
• rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.87)
• LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.87)
• Code Repair with LLMs gives an Exploration-Exploitation Tradeoff (NeurIPS 2024 · similarity 0.86)
• The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.86)