Research map / Large language models / Agents, code and math
ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
Adi Simhi, Jonathan Herzig, Martin Tutek, Itay Itzhak, Idan Szpektor, Yonatan Belinkov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add goal-oriented nudging prompt (ablation) | Gemini-2.5-Pro · ManagerBench human harm set | Harm Avoidance: 56.02 -> 0.70 | Hurt |
| add goal-oriented nudging prompt (ablation) | GPT-4o · ManagerBench human harm set | Harm Avoidance: 44.05 -> 9.45 | Hurt |
| add goal-oriented nudging prompt (ablation) | GPT-5 (high) · ManagerBench human harm set | Harm Avoidance: 87.46 -> 76.40 | Hurt |
| add goal-oriented nudging prompt (ablation) | Claude-Sonnet-4 · ManagerBench human harm set | Harm Avoidance: 95.87 -> 89.64 | Hurt |
| add goal-oriented nudging prompt (ablation) | Gemini-2.5-Pro (bounded) · ManagerBench control set | Control Pragmatism: 48.42 -> 71.44 | Improved |
| unbounded instead of bounded thinking tokens (ablation) | Gemini-2.5-Pro · ManagerBench | MB-Score: 40.16 -> 67.40 | Improved |
| increase harm percentage of the harmful option (ablation) | all models · ManagerBench human harm set | Harm Avoidance | Improved |
| raise operational benefit from 10% to 50% (ablation) | GPT-4o, Gemini, Qwen · ManagerBench human harm set | Harm Avoidance | Hurt |
| raise operational benefit from 10% to 50% (ablation) | GPT-5, Sonnet-4 · ManagerBench human harm set | Harm Avoidance | No clear effect |
| evaluate on high- vs low-perceived-harm split (ablation) | all models · ManagerBench human harm set | Harm Avoidance | Improved |
Closest papers by meaning
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.88)
- RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents (NeurIPS 2025 · similarity 0.88)
- AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents (NeurIPS 2025 · similarity 0.88)
- The Alignment Waltz: Jointly Training Agents to Collaborate for Safety (ICLR 2026 · similarity 0.88)
- CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models (ICML 2025 · similarity 0.88)
- Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
