Research map / Large language models / Agents, code and math
PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
Lehan He, Zeren Chen, Zhe Zhang, Xiang Gao, Lu Sheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reframe the same failure as a property violation instead of I/O feedback (ablation) | one-shot refinement · LiveCodeBench-Hard | pass@1: 32.0 -> 36.2 | Improved |
| select minimal-token-count counterexample instead of maximum (ablation) | PGS feedback · LiveCodeBench-Easy | pass@1 improvement: +9.1 -> +13.2 | Improved |
| select minimal-token-count counterexample instead of median (ablation) | PGS feedback · LiveCodeBench-Hard | pass@1 improvement: +3.3 -> +4.0 | Improved |
| use minimal line coverage instead of minimal token count (ablation) | PGS feedback · HumanEval | pass@1 improvement: +7.1 vs +7.0 | No clear effect |
| use maximal-complexity counterexample (max token count) rather than min: token cost (ablation) | PGS feedback · LiveCodeBench (DS-R1-32B) | avg tokens per attempt: 4.72k -> 5.68k | Hurt |
| use Qwen2.5 as Tester for DS-R1-32B generator (ablation) | PGS · LiveCodeBench | pass@1: 76.5 -> 74.9 | No clear effect |
| use Qwen3-30B as Tester for DS-R1-32B generator (ablation) | PGS · LiveCodeBench | pass@1: 76.5 -> 76.1 | No clear effect |
| inject property checks into code before refinement (ablation) | PGS · LiveCodeBench (DS-R1-32B) | Wrong Answer rate: 25.3% -> 12.6% | Improved |
| full PGS refinement after property injection (ablation) | PGS · LiveCodeBench (DS-R1-32B) | Pass rate: 64.4% -> 74.5% | Improved |
| filter properties by validation before use (ablation) | verification · LiveCodeBench-Hard | verification accuracy: 76.5% -> 88.2% | Improved |
| PGS input synthesis vs baseline TDD refinement | PGS · HumanEval (DS-V2) failed problems | fix rate: 23.1% -> 53.8% | Improved |
| PGS vs best baseline Self-Edit | PGS · LiveCodeBench (DS-R1-32B) | pass@1: 73.6 -> 76.5 | Improved |
Closest papers by meaning
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.89)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.89)
- CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules (ICLR 2024 · similarity 0.88)
- AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (ICML 2025 · similarity 0.88)
- Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
