Research map / Large language models / Instruction and fine-tuning
Reinforcement Learning with Backtracking Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLBF, which trains LLMs via SFT on BSAFE+ data plus GRPO with safety-critic feedback to emit a compact "backtrack by x tokens" signal that removes unsafe spans mid-generation.
Problem
LLMs remain vulnerable to in-context and adversarial safety violations (e.g., middle filling, GCG, decoding-parameter attacks) because safety alignment is shallow; existing reset and BSAFE backtracking mechanisms are inefficient.
Approach
A "backtrack by x tokens" control token that deletes the last x unsafe tokens without regenerating them, trained with an improved BSAFE+ SFT data pipeline (injecting violations into safe, model-generated answers) and then RL with a live LLM safety critic and a reward that penalizes unnecessary or failed backtracks.
Key results
- RLBF reduces LMSYS ASR to 1-2% across Gemma 2 2B/9B and LLaMA 3 1B/3B/8B, versus 14-17% for BSAFE+ and 17-21% for Circuit Breakers.
- On GCG attacks, RLBF achieves 4.7% ASR on AdvBench and 4.3% on HEx-PHI, versus 6.6% and 5.7% for BSAFE+.
- RLBF preserves utility: Gemma2 9B reaches 70.7% MMLU and 35.6% MATH versus IT 70.6% and 35.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RLBF instead of BSAFE+ | Gemma 2 and LLaMA 3 models · LMSYS | attack success rate (%): 14, 15, 14, 17, 16 vs 2, 2, 1, 2, 1 | Helped |
| Use RLBF instead of BSAFE+ | Gemma 2 and LLaMA 3 models · LMSYS-MF | attack success rate (%): 4, 3, 6, 5, 5 vs 5, 3, 7, 5, 3 | Mixed |
| Use RLBF instead of BSAFE+ | RLBF · AdvBench (GCG) | attack success rate (%): 6.6 -> 4.7 | Helped |
| Use RLBF instead of BSAFE+ | RLBF · HEx-PHI (GCG) | attack success rate (%): 5.7 -> 4.3 | Helped |
| Use RLBF instead of BSAFE+ | RLBF · MaliciousInstruct (Decoding Parameters) | attack success rate (%): 1.0 -> 1.0 | No clear effect |
| Use RLBF instead of BSAFE+ | RLBF · HEx-PHI (Decoding Parameters) | attack success rate (%): 5.0 -> 3.7 | Helped |
| Disable backtracking entirely (ablation) | RLBF · LMSYS | attack success rate (%): 1 -> 18 | Hurt |
| Disable backtracking in the middle of generation (ablation) | RLBF · LMSYS | attack success rate (%): 1 -> 7 | Hurt |
| Use a weaker critic (82.0% accuracy) (ablation) | RLBF · LMSYS | attack success rate (%): 1%--2% -> 2%--4% | Hurt |
| Use BSAFE-style repeat-and-edit backtracking instead of the RLBF signal (ablation) | RLBF · LMSYS | attack success rate (%): 17%--22% ASR | Hurt |
| Use RLBF instead of BSAFE+ | Gemma2 9B and LLaMA 3 8B · MMLU | solution rate (%): 70.1, 66.7 -> 70.7, 66.6 | No clear effect |
| Use RLBF instead of BSAFE+ | Gemma2 9B and LLaMA 3 8B · BBH | solution rate (%): 67.1, 63.8 -> 67.3, 64.2 | No clear effect |
Surprising findings
- Disabling backtracking only in the middle of generation still yields 7% ASR, versus 1% for full RLBF and 18% without any backtracking.
- A weaker critic (82% accuracy) still yields 2-4% ASR on LMSYS, suggesting gains come mainly from policy learning rather than critic accuracy.
Limitations
- Computational demands of backtracking and RL may challenge deployment.
- Definition of harmful content is inherently ambiguous, so the safety protocols may miss some violations.
- Critic validation uses a small manually constructed set of 100 pairs.
Experimental setup
Datasets: LMSYS MaliciousInstruct HEx-PHI AdvBench MMLU BBH GSM8K MATH
Benchmarks: LMSYS LMSYS-MF AdvBench HEx-PHI MaliciousInstruct MMLU BBH GSM8K MATH
Metrics: attack success rate solution rate attack prevention rate win rate
Baselines: IT RL Circuit Breakers BSAFE+ BSAFE
Builds on: BSAFE GRPO Circuit Breakers
Closest papers by meaning
- Backtracking Improves Generation Safety (ICLR 2025 · similarity 0.90)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.88)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.87)
- Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback (NeurIPS 2025 · similarity 0.87)
- Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.87)
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.