Research map / Large language models / Agents, code and math
SafeSearch: Automated Red-Teaming of LLM-Based Search Agents
Jianshuo Dong, Sheng Guo, Hao Wang, Xun Chen, Zhuotao Liu, Tianwei Zhang, Ke Xu, Minlie Huang, Han Qiu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch from search workflow to tool calling scaffold | GPT-4.1-mini · SafeSearch | overall ASR (%): 90.5 -> 77.8 | Improved |
| switch from search workflow to deep research scaffold | GPT-4.1-mini · SafeSearch | overall ASR (%): 90.5 -> 57.4 | Improved |
| force the tool-calling agent to use the full search budget (9 searches) (ablation) | GPT-4.1-mini tool-calling · SafeSearch | ASR (%): 74.33 -> 36.67 | Improved |
| force deep-research to use the full search budget (9 searches) (ablation) | GPT-4.1-mini deep research · SafeSearch | ASR (%): 57.00 -> 46.16 | Improved |
| increase forced search budget from 3 to 9 for tool calling (ablation) | GPT-4.1-mini tool-calling · SafeSearch | ASR (%): 49.89 -> 36.67 | Improved |
| increase auto search budget from 3 to 9 for tool calling | GPT-4.1-mini tool-calling · SafeSearch | ASR (%): 74.56 -> 74.33 | No clear effect |
| use reasoning models instead of non-reasoning ones | search agents · SafeSearch | ASR | Improved |
| use newer-generation GPT-5 / Claude-4.5 backends instead of GPT-4.1-mini | search agents (tool calling) · SafeSearch | overall ASR (%): 77.8 -> 4.6 (Claude-Sonnet-4.5) | Improved |
| provide fewer search results to the LLM (ablation) | search workflow agents · SafeSearch | ASR | Hurt |
| add reminder prompting about unreliable websites | search agents · SafeSearch | ASR | No clear effect |
| add an auxiliary filtering detector (GPT-4.1-mini) over search results | search agents · SafeSearch | ASR | Improved |
| generate test cases against stronger baseline agent (GPT-5-mini instead of Qwen3-8B) | SafeSearch test generation · SafeSearch | ASR | Improved |
Closest papers by meaning
- SafeArena: Evaluating the Safety of Autonomous Web Agents (ICML 2025 · similarity 0.88)
- AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (NeurIPS 2024 · similarity 0.87)
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.87)
- Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents (NeurIPS 2024 · similarity 0.87)
- AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.86)
- Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
