Research map / Large language models / Reasoning and chain of thought
AuPair: Golden Example Pairs for Code Repair
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AuPair, which builds a submodular-selected ordered set of golden guess-fix pairs from generated candidates and uses each as a 1-shot in-context example for code repair under an N-call inference budget.
Problem
Improving LLM code repair at inference time under a fixed compute budget, where best-of-N and self-repair plateau and produce low diversity of useful fixes.
Approach
Mine candidate (guess, fix) pairs from the model's own repair attempts, then greedily select a complementary ordered set via a submodular fix-quality-matrix procedure; each selected pair is used as a different 1-shot example per LLM call, and the best-scoring fix is returned.
Key results
- With N=144 LLM calls on CodeForces using Gemini-1.5-Pro, AuPair reaches 51.32% test pass rate and 39.73% strict accuracy, versus 16.05% and 12.04% for best-of-N at the same budget.
- AuPair pairs need about 12 pairs to match the performance of 32 random pairs, a 2.5-3x compute saving.
- AuPair pairs extracted on CodeForces transfer out-of-distribution, outperforming both baselines on all six other datasets with Gemini-1.5-Pro.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AuPair examples instead of repeated fixed-prompt sampling | Gemini-1.5-Pro · CodeForces test | test pass rate: 16.05% -> 51.32% | Helped |
| Use AuPair examples instead of self-repair feedback | Gemini-1.5-Pro · CodeForces test | test pass rate: 15.79% -> 51.32% | Helped |
| Use AuPair examples instead of repeated fixed-prompt sampling | Gemini-1.5-Flash · CodeForces test | test pass rate: 21.8% -> 37.83% | Helped |
| Use AuPair examples instead of self-repair feedback | Gemini-1.5-Flash · CodeForces test | test pass rate: 34.1% -> 37.83% | Helped |
| Select complementary AuPair examples instead of random candidate pairs (ablation) | Gemini-1.5-Pro · CodeForces | test pass rate: 12 AuPairs match the performance of 32 random pairs | Helped |
| Include repair instruction and guess/fix scores in the prompt (ablation) | AuPair repair prompting · CodeForces subset | number of problems solved | Helped |
| Use in-context repair pairs from CodeForces on other datasets | Gemini-1.5-Pro AuPair · Six out-of-distribution datasets | test pass rate | Helped |
| Use Gemini-1.5-Pro-collected AuPairs rather than AuPairs from other models (ablation) | Gemini-1.5-Pro · CodeForces | code repair performance | Helped |
| Use AuPairs rather than best-of-N generations | Gemini-1.5-Pro and Gemini-1.5-Flash · CodeForces | fix diversity | Helped |
| Use AuPairs rather than best-of-N generations | Gemma models · CodeForces | fix diversity | Hurt |
| Add AuPair repair examples to the initial guesses | Four reported models · CodeForces, difficulty levels A through F+ | test pass rate: 0.16, 0.13, 0.12, 0.09, 0.08, 0.07 | Helped |
Surprising findings
- Gemini-1.5-Pro has poor initial guesses because of input-format violations, yet AuPair yields large gains from them.
- Gemma models show lower diversity with AuPair, while Gemini models show higher diversity, and Gemma models are weaker at code repair regardless of pair source.
Limitations
- Gains depend on the model's formatting compliance and initial guess quality; pair generation requires large numbers of LLM calls.
- Evaluated only on competitive programming with grounded unit-test feedback; generalization to ungrounded domains untested.
- Gemma models show weaker code repair and reduced diversity with AuPair pairs.
Experimental setup
Datasets: CodeForces AtCoder HackerEarth CodeChef LiveCodeBench CodeJam Aizu
Benchmarks: CodeForces AtCoder HackerEarth CodeChef LiveCodeBench CodeJam Aizu
Metrics: test pass rate strict accuracy diversity score (AST subtrees)
Baselines: best-of-N self-repair random pairs
Builds on: best-of-N self-repair self-consistency
Closest papers by meaning
- Is Self-Repair a Silver Bullet for Code Generation? (ICLR 2024 · similarity 0.88)
- Code Repair with LLMs gives an Exploration-Exploitation Tradeoff (NeurIPS 2024 · similarity 0.87)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.86)
- Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization (AAAI 2025 · similarity 0.86)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.85)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.