Research map / Large language models / Language, safety and interpretability
Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models
yanjiang liu, Shuheng Zhou, Yaojie Lu, Huijia Zhu, Weiqiang Wang, Hongyu Lin, Ben He, Xianpei Han, Le Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add early-terminated exploration (ablation) | RL attack model · HarmBench test, Vicuna 7B | ASR_tst: 31.95 -> 36.54 | Improved |
| add progressive reward tracking (degrade model reward shaping) (ablation) | RL attack model · HarmBench test, Gemma 2 2b | ASR_tst: 6.15 -> 25.30 | Improved |
| combine both ETE and PRT (ablation) | RL attack model · HarmBench test, Vicuna 13B | ASR_tst: 17.80 -> 55.35 | Improved |
| add ETE and PRT (ablation) | RL attack model · HarmBench test, Llama 3 8B Instruct | ASR_tst: 14.55 -> 15.00 | No clear effect |
| add ETE and PRT (ablation) | RL attack model · HarmBench test, Gemma 2 9b Instruct | ASR_tst: 44.85 -> 44.80 | No clear effect |
| Auto-RT vs RL baseline on R2D2 | RL · HarmBench test, R2D2 | ASR_tst: few-shot 27.18; RL 8.60 -> 12.45 | Mixed |
| add early-terminated exploration (ablation) | RL attack model · HarmBench test, Vicuna 7B | DeD: 20.10 -> 43.02 | Improved |
| reward shaping with ICL-built degrade model in black-box setting | RL · HarmBench test, Llama 3 70B | ASR_tst: 4.99 -> 14.88 | Improved |
| choose degrade model as last intermediate model before the first inverse rate spike (ablation) | Progressive Reward Tracking · HarmBench test, six target models | Attack ASR | Improved |
| Auto-RT vs human-based attacks on second-round attack | AutoDAN · HarmBench, 16 models | DeD: 17.88 -> 38.19 | Improved |
| Auto-RT vs AutoDAN on first-round attack | AutoDAN · HarmBench, 16 models | ASR_tst: 55.23 vs 38.38 | Hurt |
Closest papers by meaning
- Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization (ICML 2026 · similarity 0.90)
- AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs (ICLR 2025 · similarity 0.89)
- Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks (NeurIPS 2024 · similarity 0.89)
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.89)
- MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs (AAAI 2026 · similarity 0.89)
- TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
