Research map / Reinforcement learning / RL from feedback
QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
Doyeon Lee, Eunyi Lyou, Hyunsoo Cho, Soo Kyung Kim, Joonseok Lee, Jaemoo Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GSPO with QUATRO (delta=0.001) at large k | Qwen2.5-Math-1.5B · Avg of 6 math benchmarks | Pass@256: 60.85 -> 68.68 | Improved |
| replace GSPO with QUATRO (delta=0.001) | Qwen2.5-Math-7B · Avg of 6 math benchmarks | Pass@256: 65.70 -> 73.41 | Improved |
| use smaller trust-region radius delta (0.1 -> 0.001) (ablation) | QUATRO · Avg of 6 math benchmarks, 1.5B | Pass@1: 32.63 -> 29.73 | Hurt |
| use smaller trust-region radius delta (0.1 -> 0.001) (ablation) | QUATRO · Avg of 6 math benchmarks, 1.5B | Pass@256: 63.65 -> 68.68 | Improved |
| use smaller delta (0.1 -> 0.001) (ablation) | QUATRO · Final policy entropy | entropy: 0.14 -> 0.23 | Improved |
| QUATRO vs GSPO entropy control | QUATRO · Final policy entropy | entropy: 0.05 -> 0.20 | Improved |
| replace GSPO with QUATRO to increase unique correct solutions | Qwen2.5-Math-1.5B · MATH500 | UCC@256: 26.66 -> 97.20 | Improved |
| increase inner-loop updates / learning rate | GSPO · math reasoning training | reward stability | Hurt |
| increase inner-loop updates / learning rate | QUATRO · math reasoning training | reward stability | No clear effect |
| add log-ratio stabilizer term (ablation) | QUATRO (delta=0.1, 2 inner-loop updates) · math reasoning training | reward and KL | Improved |
| replace GSPO with QUATRO on code | Qwen2.5-Coder-1.5B · HumanEval | Pass@64: 40.85 -> 50.00 | Improved |
| replace GSPO with QUATRO on code, k=1 | Qwen2.5-Coder-1.5B · MBPP zero-shot | Pass@1: 5.41 -> 6.19 | Improved |
Closest papers by meaning
- Trust-Region Adaptive Policy Optimization (ICLR 2026 · similarity 0.90)
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (ICLR 2018 · similarity 0.90)
- Simple Policy Optimization (ICML 2025 · similarity 0.88)
- Trust Region Q Adjoint Matching (NeurIPS 2026 · similarity 0.88)
- Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.87)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
