Research map / Multi-agent systems and games / Games and equilibria

The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

NeurIPS 2026 · Multi-agent systems and games › Games and equilibria · approach: Opponent modeling in competitive games · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Anchored Bipolicy Self-Play (ABS), which trains separate attacker and defender LoRA adapters on a frozen Qwen2.5 backbone to avoid the self-consistency bias of shared-parameter self-play red-teaming (Self-RedTeam).

Problem

Shared-parameter self-play red-teaming for LLM safety admits trivial or oracle Nash equilibria and suffers from self-consistency, where coupled attacker and defender gradients weaken adversarial pressure on the defender.

Approach

Freeze the base model and train role-specific LoRA adapters for attacker and defender so their gradient updates are separated, preserving adversarial pressure while keeping self-play stable and cheap.

Key results

• On Qwen2.5-3B-IT, ABS reduces WildGuardTest adversarial ASR from 0.282 (base) to 0.122, versus 0.234 for Self-RedTeam.
• In attacker-strength tests, ABS 3B and 7B attackers reach 35% and 24% ASR against the base model, versus 18% and 9% for Self-RedTeam.
• ABS adds about 1-3% parameters (50M to 250M) and trains a 14B LoRA module on three H200 GPUs in about five hours.

Surprising findings

• Fully fine-tuned Self-RedTeam Qwen2.5-3B-IT loses substantial reasoning accuracy (0.745 to 0.671 average), suggesting catastrophic forgetting.
• LoRA training sometimes collapses into repetitive output such as sequences of '!' tokens.

Limitations

• LoRA training can become unstable and collapse into nonsensical outputs.
• Larger models need careful hyperparameter tuning; ABS gains over Self-RedTeam are small at 14B.
• Theoretical results rely on assumptions such as a refusal response and reward in [-1,1], and a local trust-region neighbourhood.

Experimental setup

Datasets: WildJailbreak

Benchmarks: WildGuardTest WildJailbreak DAN HarmBench XSTest StrongREJECT GSM8K CommonSenseQA BoolQ ARC Challenge HellaSwag Harmful Behavior Dataset

Metrics: attack success rate robustness to attacks benign compliance LC winrate accuracy (pass@1)

Baselines: Self-RedTeam Qwen2.5-3B-IT Qwen2.5-7B-IT Qwen2.5-14B-IT MAGIC TriPlay-RL

Builds on: Self-RedTeam Re++ LoRA WildJailbreak

Closest papers by meaning

• AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.86)
• MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems (ICML 2026 · similarity 0.86)
• There are Levels to It: Red Teaming LLMs with Hierarchical Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
• Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.85)
• PROTEUS: A Self-Evolving Red Team with Surface Expansion for Agent Skill Ecosystems (NeurIPS 2026 · similarity 0.85)
• Learning Diverse Risk Preferences in Population-Based Self-Play (AAAI 2024 · similarity 0.85)