Research map / Reinforcement learning / RL from feedback
SWE-RM: Execution-free Feedback for Software Engineering Agents
KaShun SHUM, Binyuan Hui, Jiawei Chen, Lei Zhang, X. W., Jiaxi Yang, Yuzhen Huang, Junyang Lin, Junxian He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 2:1 positive-to-negative ratio instead of 1:1 (ablation) | SWE-RM verifier (Qwen3-Coder-Flash) · SWE-bench Verified | AUC: 0.782 -> 0.805 | Improved |
| use 2:1 positive-to-negative ratio instead of 1:8 (ablation) | SWE-RM verifier (Qwen3-Coder-Flash) · SWE-bench Verified | ECE: 0.349 -> 0.080 | Improved |
| use 2:1 ratio instead of 1:1 (ablation) | SWE-RM verifier (Qwen3-Coder-Max) · SWE-bench Verified | RM@32: 70.2% -> 71.0% | Mixed |
| extend verifier context length from 16k to 256k (ablation) | SWE-RM verifier · SWE-bench Verified | RM@32: 66.8% -> 74.4% | Improved |
| extend context length from 16k to 256k (ablation) | SWE-RM verifier · SWE-bench Verified | score rate: 0.5% -> 100% | Improved |
| use mix-policy data instead of on-policy data (ablation) | SWE-RM verifier (Qwen3-Coder-Flash) · SWE-bench Verified | ECE: 0.148 -> 0.033 | Improved |
| use mix-policy instead of on-policy data (ablation) | SWE-RM verifier (Qwen3-Coder-Max) · SWE-bench Verified | RM@32: 71.0 -> 70.2 | Hurt |
| train on SWE-rebench instead of SWE-smith (ablation) | SWE-RM verifier (Qwen3-Coder-Flash) · SWE-bench Verified | AUC: 0.781 -> 0.814 | Improved |
| add SWE-smith to SWE-Gym training data (ablation) | SWE-RM verifier (Qwen3-Coder-Flash) · SWE-bench Verified | ECE: 0.087 -> 0.034 | Improved |
| use SWE-RM execution-free verifier instead of DeepSWE EF verifier | verifier · SWE-bench Verified (Qwen3-Coder-Flash) | RM@32: 53.2% -> 62.0% | Improved |
| use hybrid feedback instead of execution-based only | RL on Qwen3-30B-A3B · SWE-bench Verified | pass@1: 51.8% -> 54.8% | Improved |
| use hybrid feedback instead of execution-free only (ablation) | RL on Qwen3-30B-A3B · Terminal Bench | score: 31.3 -> 32.5 | Improved |
Closest papers by meaning
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.90)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.89)
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.88)
- CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models (NeurIPS 2026 · similarity 0.88)
- SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution (NeurIPS 2025 · similarity 0.88)
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
