Research map / Multi-agent systems and games / LLM agents

The Alignment Waltz: Jointly Training Agents to Collaborate for Safety

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WaltzRL, a multi-agent RL framework where a conversation agent and a feedback agent are jointly trained as a positive-sum game to reduce both unsafe responses and overrefusals, using a Dynamic Improvement Reward for the feedback agent.

Problem

LLM safety alignment trades off unsafe responses against overrefusal of benign prompts; standalone safeguard models block whole responses, which increases overrefusal.

Approach

Treat safety alignment as a positive-sum collaborative game: a feedback agent is trained with a Dynamic Improvement Reward (change in conversation agent reward after incorporating its feedback), and both agents are updated jointly with multi-agent REINFORCE++, with a two-stage schedule and adaptive feedback triggering at inference.

Key results

• WaltzRL reduces average ASR to 3.7% (vs 26.5% baseline response) and average ORR to 7.6% (vs 25.7%), outperforming Llama Guard 4 safeguard and single-model RL.
• On WildJailbreak, ASR falls from 39.0% (baseline) to 4.6% with WaltzRL; on OR-Bench, ORR falls from 45.3% to 9.9%.
• WaltzRL training causes little degradation on AlpacaEval LCWR (37.2 to 35.9), IFEval and MMLU (68.0 to 68.1), and feedback trigger rate on AlpacaEval is only 6.7%.

Surprising findings

• Safeguards increase overrefusal most when applied on top of a model that already has low overrefusal (ORR 8.6% to 14.9% after safeguard on single-model RL).
• Oracle label-converted template feedback underperforms WaltzRL's detailed feedback, especially for reducing overrefusal, suggesting rationales are needed to get the model to comply.

Limitations

• Experiments are only on English datasets.
• Only a single feedback round (T_max=1) was tested due to compute constraints.
• Evaluated only on static adversarial prompts, not adaptive attacks; overrefusal and unsafe rates are not fully eliminated.

Experimental setup

Datasets: WildJailbreak OR-Bench-80K OR-Bench-Hard-1K FORTRESS StrongREJECT AlpacaEval 2.0 IFEval GPQA Diamond MMLU TruthfulQA

Benchmarks: WildJailbreak FORTRESS StrongREJECT OR-Bench-Hard-1K FORTRESS benign AlpacaEval 2.0 IFEval GPQA Diamond MMLU TruthfulQA

Metrics: Attack Success Rate Over-Refuse Rate Feedback Trigger Rate Label accuracy length-controlled win rate accuracy F1 score

Baselines: Llama Guard 4 safeguard Single-model RL Inference-time collaboration Oracle label-converted feedback

Builds on: REINFORCE++ GRPO PPO Llama Guard WildJailbreak

Closest papers by meaning

• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning (ICLR 2026 · similarity 0.90)
• Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction (ICML 2026 · similarity 0.88)
• Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization (NeurIPS 2025 · similarity 0.88)
• RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates (ICML 2026 · similarity 0.88)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.88)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.88)