Research map / Large language models / Alignment and preferences
On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Qwen3-8B with OORL, combining on-policy REINFORCE++ using unit-test binary rewards on code translation with a group-based preference loss (GEPO) over equivalent vs. inequivalent LLVM IR groups, improving multi-language code generation and translation.
Problem
Code LLMs perform much worse on less-common programming languages than on Python/C++, and coarse rule-based rewards give no process-level signal about functional equivalence of code across languages.
Approach
Combine on-policy RL with a binary unit-test reward on code translation with GEPO, a group-wise preference loss that contrasts the average reward of a winner group of functionally equivalent LLVM IRs (from different optimization levels) against a loser group of inequivalent IRs, plus a variance penalty that pulls winners together.
Key results
- Qwen3-8B-OORL achieves 76.31 average pass@1 on MultiPL-E, versus 65.07 for Qwen3-8B and 68.71 for Qwen2.5-Coder-7B-Inst.
- On CrossPLEval, Qwen3-8B-OORL averages 63.15 with REINFORCE++ plus GEPO, versus 50.96 for base Qwen3-8B and 58.70 with REINFORCE++ plus DPO.
- Ablation shows REINFORCE++ alone lifts MultiPL-E from 65.06 to 73.60; adding GEPO reaches 76.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Qwen3-8B with OORL and code translation | Qwen3-8B · MultiPL-E | pass@1 (Avg.): 65.07 -> 76.31 | Helped |
| Train with Python-to-Haskell code translation using OORL | Qwen3-8B · CrossPLEval, Python to Haskell | pass@1: 15.13 -> 28.48 | Helped |
| Add on-policy REINFORCE++ with code translation rewards (ablation) | Qwen3-8B · MultiPL-E | score: 65.06 -> 73.60 | Helped |
| Add on-policy REINFORCE++ with code translation rewards (ablation) | Qwen3-8B · CrossPLEval | Avg. score: 50.96 -> 58.46 | Helped |
| Add DPO to REINFORCE++ (ablation) | Qwen3-8B · MultiPL-E | score: 73.60 -> 73.48 | Hurt |
| Add DPO to REINFORCE++ (ablation) | Qwen3-8B · CrossPLEval | Avg. score: 58.46 -> 58.70 | Helped |
| Add GEPO to REINFORCE++ instead of DPO (ablation) | Qwen3-8B · MultiPL-E | score: 73.60 -> 76.31 | Helped |
| Add GEPO to REINFORCE++ instead of DPO (ablation) | Qwen3-8B · CrossPLEval | Avg. score: 58.46 -> 63.15 | Helped |
Surprising findings
- DPO added on top of REINFORCE++ gives only a marginal CrossPLEval gain (58.46 to 58.70) and slightly lowers MultiPL-E (73.60 to 73.48).
Limitations
- GEPO processes grouped responses per prompt, requiring more GPU memory than pairwise methods.
- Serializing group processing to save memory increases training latency.
- CrossPLEval is small (119 problems) and manually constructed, so results may not generalize widely.
Experimental setup
Datasets: SYNTHETIC-1 APPS SLTrans TACO MultiPL-E
Benchmarks: MultiPL-E CrossPLEval
Metrics: pass@1
Baselines: DS-Coder-V2-Lite-Inst Qwen2.5-Coder-7B-Inst Qwen3-8B DPO
Builds on: REINFORCE++ DPO GRPO IRCoder
Closest papers by meaning
- Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization (NeurIPS 2025 · similarity 0.89)
- R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning (ICLR 2026 · similarity 0.88)
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.88)
- $\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis (ICLR 2024 · similarity 0.87)
- CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models (NeurIPS 2026 · similarity 0.87)
- Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.