Research map / Large language models / Instruction and fine-tuning
Exploration Hacking: Can LLMs Learn to Resist RL Training?
Eyon Jang, Damon Falck, Joschka Braun, Nathalie Maria Kirch, Achyutha Menon, Perusha Moodley, Scott Emmons, Roland S. Zimmermann, David Lindner
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune (RL-lock) the model instead of only prompting it to underperform (ablation) | Qwen3-14B · KernelBench RL elicitation | resistance to elicitation: recovers within 100 steps -> suppressed for over 500 steps | Improved |
| add few-shot (3-shot) examples to prompt locking (ablation) | Qwen3-14B · KernelBench | resistance to RL elicitation | Improved |
| add more shots to prompt locking (ablation) | Qwen3-14B · BigCodeBench | resistance to RL elicitation | No clear effect |
| use deterministic shortest-answer strategy instead of random wrong answer (ablation) | RL-locked Qwen3-14B · WMDP-Bio | accuracy under RL elicitation: 25% -> over 50% (wrong answer) vs flat (shortest answer) | Improved |
| increase locking strength (80% vs 95% reliability) (ablation) | RL-locked Qwen3-14B · WMDP-Bio | resistance to elicitation | No clear effect |
| use stronger SFT locking (180 vs 40 steps) (ablation) | SFT-locked Qwen3-14B · BigCodeBench RL elicitation | malign performance | Improved |
| use continuous multi-test-case reward rather than multiple choice | RL elicitation · KernelBench, BigCodeBench vs WMDP-Bio | difficulty of resisting elicitation | Hurt |
| add action-only or CoT monitor (gpt-5-mini) | exploration hacking model organisms · BigCodeBench | detection ROC-AUC | Improved |
| add Gaussian weight noise to LoRA adapters | BigCodeBench model organism · BigCodeBench with trigger | accuracy: transient improvement for sigma <= 0.02, then collapse | Mixed |
| add SFT on benign traces | locked model organism · WMDP-Bio [eval] segment | accuracy: 80.0% -> 98.0% | Improved |
| add SFT on filtered Qwen3-4B traces | locked model organism · BigCodeBench | malign pass@1 | Improved |
| increase scaffolding from minimal to full in system prompt (ablation) | frontier models · WMDP/VCT audit | exploration hacking rate | Improved |
Closest papers by meaning
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.90)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.89)
- LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.89)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.88)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
