Research map / Multi-agent systems and games / LLM agents

Building a Foundational Guardrail for General Agentic Systems via Synthetic Data

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AuraGen, a synthetic risky-trajectory engine, plus the Safiron pre-execution guardrail (adapter + SFT/GRPO-trained Ministral-8B guardian) and the human-verified Pre-Exec Bench for plan-level safety.

Problem

Guardrails for LLM agents mostly act post-execution, and there is little training data, guardian models, or evaluation benchmarks for detecting risks in agent plans before any action is executed.

Approach

Three-stage synthetic pipeline (benign trajectory synthesis, risk injection with four strategies, reward-model quality filtering) generates training data; a compact guardian (Safiron) is trained with SFT then GRPO using a reward for harmless/harmful correctness and risk category; a cross-planner adapter normalizes input formats; a human-verified benchmark evaluates plan-level detection.

Key results

• Safiron (SFT+GRPO) reaches 0.973 harmful detection precision, 0.646 risk category accuracy and 0.570 explanation correctness on Pre-Exec Bench, versus 0.318 risk category accuracy for Claude-3.7-Sonnet.
• Pre-Exec Bench contains 1,001 harmless and 671 risky samples built with human verification.
• Llama-Guard-3-8B and LlamaFirewall baselines both score below 60% classification accuracy on Pre-Exec Bench.

Surprising findings

• GPT-5 reaches 0.990 harmful detection precision but only 0.425 classification accuracy, indicating over-flagging of benign plans.
• The harmless:harmful ratio affects performance more than training set size, with 2k-to-10k scaling giving only small gains.

Limitations

• Case study uses only 50 harmful and 50 harmless human-edited trajectories per system.
• Explanation quality is not optimized directly in the RL reward and is judged by an LLM-based judge.

Experimental setup

Datasets: AuraGen synthetic trajectories AgentSafetyBench scenarios

Benchmarks: Pre-Exec Bench

Metrics: classification accuracy harmful detection precision risk category accuracy explanation correctness

Baselines: GPT-5 GPT-4o Claude-3.7-Sonnet Gemini-2.5-Pro DeepSeek-V3 Qwen2.5-72B Llama-Guard-3-8B LlamaFirewall

Builds on: GRPO Llama Guard AgentSafetyBench R-Judge LlamaFirewall

Closest papers by meaning

• PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety (NeurIPS 2026 · similarity 0.89)
• AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.88)
• Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correction (ICML 2026 · similarity 0.88)
• LiSA: Lifelong Safety Adaptation via Conservative Policy Induction (NeurIPS 2026 · similarity 0.88)
• Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents (ICML 2026 · similarity 0.88)
• Constitutional Black-Box Monitoring for Scheming in LLM Agents (ICML 2026 · similarity 0.88)