Research map / Large language models / Agents, code and math
Programming Puzzles
Tal Schuster, Ashwin Kalyan, Alex Polozov, Adam Tauman Kalai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add English descriptions to the Medium prompt | Codex davinci · P3 v0.2 (397 problems) | pass@1: 11.2% -> 18.3% | Improved |
| Add English descriptions to the Medium prompt | Codex cushman · P3 v0.2 (397 problems) | pass@1: 7.1% -> 14.9% | Improved |
| Use the larger davinci engine instead of cushman | Codex with Long prompt · P3 v0.2 (397 problems) | pass@1,000: 76.5% -> 79.8% | Improved |
| Train a random forest enumerative solver from discovered solutions instead of using uniform search | Enumerative solver · P3 v0.1, 138 grammar-supported problems | problems solved: 28% more than the uniform solver | Improved |
| Train a Transformer enumerative solver from discovered solutions instead of using uniform search | Enumerative solver · P3 v0.1, 138 grammar-supported problems | problems solved: 43% more than the uniform solver | Improved |
| Bootstrap GPT-3 prompts with previously found solutions | GPT-3 solver · P3 v0.1, 138 problems | problems solved | Improved |
| Bootstrap Codex prompts with previously found solutions | Codex solver · P3 v0.1, 138 problems | problems solved | No clear effect |
| Use the Long prompt instead of the Medium prompt | Codex davinci · P3 v0.2, macro-average across domains | score: 51.2 -> 54.8 | Improved |
| Use the Long prompt instead of the Medium prompt | Codex cushman · P3 v0.2, macro-average across domains | score: 44.2 -> 49.8 | Improved |
Closest papers by meaning
- ACES: Generating a Diversity of Challenging Programming Puzzles with Autotelic Generative Models (NeurIPS 2024 · similarity 0.89)
- Learning to Reason via Program Generation, Emulation, and Search (NeurIPS 2024 · similarity 0.85)
- Logic.py: Bridging the Gap between LLMs and Constraint Solvers (NeurIPS 2025 · similarity 0.84)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.84)
- FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale (NeurIPS 2026 · similarity 0.84)
- CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
