Research map / Large language models / Reasoning and chain of thought
PADA-Coder: Improving Plan-Following Code Generation via Perturbation-Verified Attention Distillation and Dynamic Alignment
Yihong Huang, Ke Qin, Rongzheng Wang, Muquan Li, Jiakai Li, Xiurui Xie, Shuang Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PADA-Coder, which improves plan-following code generation in LLMs by distilling attention toward plan-relevant tokens, verified via perturbation, with dynamic alignment.
Problem
In Plan-then-Code pipelines, adding the plan and verification report enlarges context, so the model's attention drifts to irrelevant tokens and recent code, causing it to overlook critical constraints from the plan.
Approach
Distills attention patterns into the model so it attends to plan-relevant tokens, using perturbation to verify which attention targets matter, combined with dynamic alignment during generation (based on abstract only; details unavailable).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add attention alignment training (ablation) | PADA (Qwen3-4B) · APPS | Pass@1: 57.8 -> 65.1 | Improved |
| add attention alignment training (ablation) | PADA (Qwen3-4B) · HumanEval | Pass@1: 93.9 -> 96.8 | Improved |
| add progress-aware sliding window (ablation) | PADA (Qwen3-4B) · APPS | Pass@1: 61.8 -> 65.1 | Improved |
| add progress-aware sliding window (ablation) | PADA (Qwen3-4B) · HumanEval | Pass@1: 93.2 -> 96.8 | Improved |
| add perturbation-verified key token selection (ablation) | PADA (Qwen3-4B) · APPS | Pass@1: 63.1 -> 65.1 | Improved |
| add perturbation-verified key token selection (ablation) | PADA (Qwen3-4B) · HumanEval | Pass@1: 95.1 -> 96.8 | Improved |
| add difficulty-aware gating (ablation) | PADA (Qwen3-4B) · APPS | Pass@1: 65.1 -> 65.1 | No clear effect |
| add difficulty-aware gating (ablation) | PADA (Qwen3-4B) · HumanEval | Pass@1: 96.3 -> 96.8 | No clear effect |
| use Qwen3-32B as teacher instead of none (ablation) | PADA (Qwen2.5-Coder-7B) · APPS | Pass@1: 58.1 -> 61.3 | Improved |
| use similar-scale Qwen3-4B as teacher (ablation) | PADA (Qwen2.5-Coder-7B) · APPS | Pass@1: 58.1 -> 58.4 | No clear effect |
| use GPT-5.2 prompted key tokens as teacher (ablation) | PADA (Qwen2.5-Coder-7B) · HumanEval, MBPP+, APPS | Pass@1: 58.1 -> 58.8 | Mixed |
| add teacher model to PADA (ablation) | PADA (Qwen3-4B) · Code benchmarks average (7 benchmarks) | Pass@1 average: 75.3 -> 77.8 | Improved |
Experimental setup
Builds on: plan-then-code
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.86)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.86)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.85)
- CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance (ICML 2025 · similarity 0.85)
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.85)
- rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
