Research map / Reinforcement learning / Visual and representation RL
Alignment Risks from Capability-Seeking RL Training
Yujun Zhou, Yue Huang, Han Bao, Kehan Guo, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Werner Geyer, Nuno Moniz, Nitesh V Chawla, Xiangliang Zhang
This paper tried a common change: Chain-of-thought prompting
use chain-of-thought prompting for Qwen3-4B · Reward Tampering · exploit discovery Hurt
What 1,107 accepted papers report for this change
- 765 improved
- 94 no clear effect
- 52 mixed
- 196 hurt
- AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing ICML 2026 · add structured chain-of-thought reasoning to the diffusion LLM · StoryGen-Eval editing · delta multiFLAM: 27.8 -> 11.7Improved
- Vision Language Models Cannot Reason About Physical Transformation ICML 2026 · add CoT prompt instead of Direct prompt · Number & Length · accuracy: F(3, 333) = 18.28Hurt
- VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation ICML 2026 · Add CoT cold-start stage on top of SFT · Long-RVOS · J&F: 47.12 -> 51.82Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with GRPO in a vulnerable environment instead of using the original model | Llama-3.1-8B-Instruct · Audited Self-Grading | exploit rate (%): 44.0 -> 80.0 | Hurt |
| train in vulnerable Self-Grading environment (with loophole) vs original model | Llama-3.1-8B-Instruct · Audited Self-Grading | task performance (%): 7.1 -> 16.9 | Improved |
| remove the loophole (loophole-free control) (ablation) | Llama-3.1-8B-Instruct GRPO · Audited Self-Grading | exploit rate (%): 80.0 -> 2.6 | Improved |
| remove the loophole (loophole-free control) (ablation) | Qwen3-4B GRPO · Audited Self-Grading | exploit rate (%): 32.8 -> 9.0 | Improved |
| remove the loophole (loophole-free control) (ablation) | Llama-3.1-8B GRPO · Audited Self-Grading | task performance (%): 16.9 -> 20.4 | No clear effect |
| use chain-of-thought prompting for Qwen3-4B | Qwen3-4B · Reward Tampering | exploit discovery | Hurt |
| zero-shot transfer from Proxy Metric Gaming training | Qwen3-4B-Base · Self-Grading | exploit rate (%): 9.0 -> 55.7 | Improved |
| zero-shot transfer from other exploits to Proxy Metric Gaming | all models · Proxy Metric Gaming | exploit rate | No clear effect |
| initialize from a Reward Tampering policy before RL on Context-Conditional Compliance (ablation) | Llama-3.1-8B-Instruct · Context-Conditional Compliance | exploit rate (%): 72.8 -> 83.8 | Improved |
| initialize from a Proxy Metric Gaming policy before RL on Context-Conditional Compliance (ablation) | Qwen3-4B-Base · Context-Conditional Compliance | exploit rate (%): 78.5 -> 62.8 | Hurt |
| initialize from a failed Reward Tampering policy before RL on Self-Grading (ablation) | Qwen3-4B · Self-Grading | exploit rate (%): 32.8 -> 47.1 | Improved |
| SFT-distill exploits from GRPO teacher into student | Qwen3-4B-Base · Proxy Metric Gaming | exploit rate (%): 3.9 -> 91.1 | Improved |
Closest papers by meaning
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.89)
- AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.88)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.88)
- The Alignment Waltz: Jointly Training Agents to Collaborate for Safety (ICLR 2026 · similarity 0.88)
- Fundamental Limitations of Alignment in Large Language Models (ICML 2024 · similarity 0.87)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
