Research map / Large language models / Agents, code and math
ACES: Generating a Diversity of Challenging Programming Puzzles with Autotelic Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACES, an autotelic goal-directed LLM search that iteratively generates diverse, difficult Python programming puzzles targeting skill-combination niches, using empirical solver difficulty as fitness.
Problem
Automatically generating a diverse set of challenging yet solvable programming puzzles, where difficulty is calibrated to LLM solvers, to produce harder and more diverse benchmarks than existing code evaluations.
Approach
ACES combines MAP-Elites-style quality-diversity archiving over LLM-labeled skill-combination niches with goal-directed prompting: it samples a target niche and difficult neighboring puzzles as in-context examples, and uses the empirical success rate of a solver LLM (Llama-3-70B) as the fitness.
Key results
- ACES variants fill up to ~700 skill-combination niches and reach higher QD-score than ELM, ELM-CVT and StaticGen baselines across 3 seeds.
- Best solver Mixtral-8x22B-Instruct-v0.1 achieves only 47.3% pass@1 on ACES-generated problems versus 78.7% for CodeQwen1.5-7B-Chat on HumanEval+; generated puzzles are about three times more challenging on average across 11 code LLMs.
- Llama-3-8B finetuned on ACES-ELM data reaches 53.3 pass@1 on the hardest test set, outperforming Llama-3-70B and WizardCoder/StaticGen-finetuned models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama-3-8B on ACES-ELM-generated data instead of WizardCoder-generated data | Llama-3-8B · Most challenging test set | pass@1: 49.4 to 53.3 | Helped |
| Fine-tune Llama-3-8B on ACES-ELM-generated data instead of StaticGen-generated data | Llama-3-8B · Most challenging test set | pass@1: 41.6 to 53.3 | Helped |
| Generate puzzles with ACES-ELM instead of using HumanEval+ | CodeQwen1.5-7B-Chat · HumanEval+ vs. ACES-ELM generated puzzles | pass@1: 78.7 to 15% | Hurt |
| Generate puzzles with ACES-ELM instead of using HumanEval+ | Meta-Llama-3-70B-Instruct · HumanEval+ vs. ACES-ELM generated puzzles | pass@1: 72 to 47% | Hurt |
| Generate puzzles with ACES-ELM instead of using HumanEval+ | Mixtral-8x22B-Instruct-v0.1 · HumanEval+ vs. ACES-ELM generated puzzles | pass@1: 72 to 34% | Hurt |
| Scale ACES-ELM from Llama-3-70B to Mistral Large 2 | ACES-ELM · Generated puzzle archive | QD score: up to 25.6% | Helped |
| Scale ACES-ELM from Llama-3-70B to Llama 3 405B | ACES-ELM · Generated puzzle archive | QD score: 12.3% | Helped |
| Use goal-directed generation rather than undirected mutation (ablation) | Puzzle generation with semantic skill descriptors · Programming puzzle generation | semantic diversity | Helped |
| Use goal-directed generation rather than undirected mutation (ablation) | Puzzle generation · Programming puzzle generation | difficulty | Helped |
| Replace semantic skill descriptors with embedding-space CVT cells (ablation) | ELM-CVT · Programming puzzle generation | embedding diversity | Mixed |
| Use static few-shot examples rather than reusing generated puzzles (ablation) | StaticGen · Programming puzzle generation | difficulty | Hurt |
| Use Llama-3-70B ACES-ELM puzzles rather than HumanEval | Generated benchmark · Correlation with LiveCodeBench | correlation: 0.46 to 0.83 | Helped |
Surprising findings
- Generated-benchmark difficulty correlates more with LiveCodeBench (up to 0.83) than HumanEval does (0.46), suggesting HumanEval results may reflect contamination.
- Labeling with the generator in the same prompt led to label hacking, so descriptions were generated separately from puzzles.
Limitations
- Relies on assumptions about labeling accuracy, generator goal-reaching ability, and parent-child heredity; labeler precision 0.71 and recall 0.75.
- Tests solutions with only one set of arguments, so solution robustness is not measured.
- Empirical difficulty is expensive to compute (50 solver attempts per puzzle) and puzzles may not be hard for good reasons.
Experimental setup
Datasets: P3 train P3 test HumanEval+ BigCodeBench LiveCodeBench
Benchmarks: HumanEval+ BigCodeBench LiveCodeBench P3 test
Metrics: greedy pass@1 semantic diversity (niches filled) embedding diversity fitness QD-score
Baselines: ELM ELM-CVT StaticGen RandomGen ACES-ELM WizardCoder
Builds on: MAP-Elites ELM P3 (Python Programming Puzzles) Skill-Mix Unnatural Instructions
Closest papers by meaning
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.86)
- Planning in Natural Language Improves LLM Search for Code Generation (ICLR 2025 · similarity 0.85)
- Toward Interactive Understanding of Code APIs (NeurIPS 2026 · similarity 0.85)
- Logic.py: Bridging the Gap between LLMs and Constraint Solvers (NeurIPS 2025 · similarity 0.85)
- Learning to Reason via Program Generation, Emulation, and Search (NeurIPS 2024 · similarity 0.85)
- FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.