Research map / Large language models / Agents, code and math
Training Language Models to Generate Quality Code with Program Analysis Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReaL, a PPO-based RL framework that rewards LLM code generation with a hybrid of program-analysis quality checks (security and MyPy-based maintainability) and unit-test functionality, evaluated on SecCodePLT+, SafeSQL, and APPS+ with Qwen2.5-Coder-Instruct at 0.5B-7B.
Problem
LLM code generation often produces code that is functionally plausible but insecure (e.g., SQL injection) or poorly maintainable (e.g., missing type annotations); existing fixes rely on costly annotations or brittle rule-based heuristics, and evaluations treat functionality and quality separately.
Approach
Use automated program analysis (an information-flow-style security detector and MyPy-based maintainability checks) as a verifiable, reference-free reward combined with unit-test pass rate in PPO, so the model learns quality code without human-written safe references or per-task rules.
Key results
- On SafeSQL at 7B, ReaL reaches 0.8588 joint Func.-Qual. pass rate versus 0.7294 for the best SFT baseline and 0.5882 for vanilla.
- On APPS+ at 7B, ReaL achieves 0.6204 joint score versus 0.4663 for SFT, with quality rising from 0.1580 (vanilla) to 0.9229.
- At 0.5B on SecCodePLT+, SFT slightly outperforms ReaL on joint score (0.4573 vs 0.3963), which the authors attribute to limited base model capacity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the hybrid functionality-plus-quality reward instead of SFT | ReaL, Qwen2.5-Coder-Instruct 0.5B · SecCodePLT+ | Func.-Qual. pass rate: 0.4573 -> 0.3963 | Hurt |
| Use the hybrid functionality-plus-quality reward instead of SFT | ReaL · SafeSQL, APPS+; 0.5B, 3B, and 7B | Func.-Qual. pass rate: SafeSQL 0.5527 -> 0.6941; 0.6824 -> 0.8471; 0.7294 -> 0.8588; APPS+ 0.1888 -> 0.2909; 0.4046 -> 0.5241; 0.4663 -> 0.6204 | Helped |
| Use the hybrid functionality-plus-quality reward rather than functionality-only reward (ablation) | ReaL · SecCodePLT+, SafeSQL, and APPS+; 0.5B, 3B, and 7B | Func.-Qual. pass rate: SecCodePLT+ 0.0854 -> 0.3963; 0.2073 -> 0.6037; 0.2866 -> 0.6159; SafeSQL 0.0235 -> 0.6941; 0.0588 -> 0.8471; 0.0353 -> | Helped |
| Add functionality reward to quality-only reward (ablation) | ReaL · SecCodePLT+, SafeSQL, and APPS+; 0.5B, 3B, and 7B | Func.-Qual. pass rate: SecCodePLT+ 0.0732 -> 0.3963; 0.1220 -> 0.6037; 0.2866 -> 0.6159; SafeSQL 0.3529 -> 0.6941; 0.5529 -> 0.8471; 0.6353 -> | Mixed |
| Use program-analysis detector rewards instead of safety unit-test rewards (ablation) | ReaL · SecCodePLT+ and SafeSQL; 0.5B, 3B, and 7B | Func.-Qual. pass rate: SecCodePLT+ 0.1341 -> 0.3963; 0.2439 -> 0.6037; 0.3171 -> 0.6159; SafeSQL 0.3882 -> 0.6941; 0.7412 -> 0.8471; 0.7647 -> | Helped |
Surprising findings
- Quality-only reward causes reward hacking, with the model producing trivial outputs such as empty code or comments.
- The authors found CodeQL unexpectedly ineffective on SecCodePLT+, motivating their own detector.
Limitations
- Vulnerability detectors rely on heuristic approximations and do not cover the full breadth of CWE types (18 CWEs targeted).
- At 0.5B scale, SFT beats ReaL on SecCodePLT+ joint metric, suggesting RL needs a reasonably strong base model.
- Hybrid reward weighting alpha trades off quality and functionality and individual dimensions are slightly lower than single-reward variants.
Experimental setup
Datasets: SecCodePLT+ SafeSQL APPS+ PurpleLLaMA
Benchmarks: SecCodePLT+ SafeSQL APPS+
Metrics: Function pass rate Quality pass rate Func.-Qual. joint pass rate
Baselines: Vanilla SafeCoder SVEN CodeGuard+ PromSec SFT PromptEng Safety unit tests reward
Builds on: PPO Generalized Advantage Estimation SecCodePLT APPS MyPy
Closest papers by meaning
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.90)
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.89)
- Rethinking Verification for LLM Code Generation: From Generation to Testing (NeurIPS 2025 · similarity 0.89)
- Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.88)
- NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · similarity 0.88)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.