Research map / Large language models / Language, safety and interpretability
Is Programming by Example Solved by LLMs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pretrained code LLMs are weak at programming-by-example, but fine-tuning on synthetic (program, I/O) data from a seed set, plus a wake-sleep style adaptation loop, gives strong PBE over list, string, and LOGO graphics domains.
Problem
Whether LLMs can solve programming-by-examples (inferring programs from input-output examples alone), which has so far been dominated by domain-specific languages and symbolic search, and how well they generalize out of distribution.
Approach
Generate a large synthetic dataset of (program, inputs, outputs) by prompting an LLM with a small seed set and executing programs to get outputs, fine-tune a code LLM to infer programs from I/O, then adapt to new distributions with a wake-sleep style loop that adds solved unlabeled problems back to the seed set.
Key results
- The 33B fine-tuned model reaches 82% generalization accuracy on the string PBE benchmark versus 33% for FlashFill, with 88% oracle accuracy.
- On LOGO graphics the fine-tuned model solves 90% of the test set, surpassing DreamCoder, LILO and Regal, which use natural-language descriptions.
- Adaptation via wake-sleep raises out-of-distribution accuracy by around 10% or more in all domains (about 16% relative for text/list and about 190% relative for LOGO).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune DeepSeekCoder on synthetic PBE examples | DeepSeekCoder for text editing · Text editing (holdout problems) | generalization accuracy: 33% -> 82% | Helped |
| Fine-tune DeepSeekCoder on synthetic PBE examples | DeepSeekCoder for LOGO graphics · LOGO test set | accuracy | Helped |
| Fine-tune on list-function PBE examples | DeepSeekCoder for list-to-list synthesis · λ² list-to-list benchmark | accuracy | Helped |
| Use SyGuS rather than PROSE problems for fine-tuning | DeepSeekCoder-7B · PROSE | accuracy: 76% -> 59% | Hurt |
| Adapt the fine-tuned model using unlabeled test-distribution examples and accepted solutions | Fine-tuned PBE models · Out-of-distribution list, text, and LOGO tasks | accuracy: around 10% or more | Helped |
| Adapt the fine-tuned model using unlabeled test-distribution examples and accepted solutions | Fine-tuned PBE models · Out-of-distribution text and list tasks | relative accuracy increase: about 16% | Helped |
| Adapt the fine-tuned model using unlabeled test-distribution examples and accepted solutions | Fine-tuned PBE model · Out-of-distribution LOGO tasks | relative accuracy increase: about 190% | Helped |
Surprising findings
- Compute needed to solve a problem is better predicted by posterior description length than by program size or prior description length, suggesting the model is not blind guess-and-check.
- Pretrained and instruction-tuned models are poor PBE systems before fine-tuning.
Limitations
- Using 7B-33B models for PBE is impractical for most end users running on laptops.
- Out-of-distribution performance degrades, and adaptation does not solve problems very unlike those already solved.
- Pretrained models rely on sampling many programs; the approach is rejection sampling rather than efficient search.
Experimental setup
Datasets: Rule et al. 2024 list functions Shi et al. 2023 LambdaBeam PROSE / FlashFill++ string problems SyGuS string problems Regal/DreamCoder LOGO programs lambda^2 list benchmark
Benchmarks: list functions (Rule et al. 2024) LambdaBeam (Shi et al. 2023) PROSE string benchmark SyGuS LOGO graphics test set lambda^2 list problems
Metrics: generalization accuracy oracle accuracy fraction of problems solved
Baselines: GPT-4 GPT-4o FlashFill FlashFill++ DreamCoder LILO Regal Metagol RobustFill Fleet lambda^2
Builds on: self-instruct wake-sleep DreamCoder FlashFill
Closest papers by meaning
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.88)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.87)
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.86)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.86)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.86)
- Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.