Research map / Learning theory and algorithms / Combinatorial and search optimization
Combining Cost Constrained Runtime Monitors for AI Safety
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an exhaustive-search-plus-Neyman-Pearson procedure for combining costly runtime monitors under a unified cost budget, and shows on backdoored-code detection that two Llama monitors can Pareto-dominate either alone.
Problem
Runtime monitors for AI outputs are costly to run and interventions (audits) are costly on false positives, so developers need a principled way to decide when and which monitors to call and which outputs to audit, maximizing recall under an average-case budget.
Approach
Decompose policy optimization into choosing monitor-invocation strategies (searched exhaustively over a restricted class) and allocating audits by likelihood ratio via the Neyman-Pearson lemma; trade off monitoring and auditing spend through stochastic monitor calls under a unified budget.
Key results
- Optimal stochastic policy Pi_NPS more than doubles recall versus deterministic policies at a $1.20 budget with audit cost 20x monitor cost (calls monitor 71% of the time, 2.4% FPR vs 1%).
- Two-monitor policy Pi_SR (Llama 8B first, Llama 70B called in score region [5, 6.2)) never does worse than single-monitor or always-both baselines and achieves higher safety at some budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a likelihood-ratio cutoff instead of auditing the highest raw monitor scores | Single-monitor monitoring with Llama 3.1 70B · BigCodeBench backdoor detection | recall | Helped |
| Stochastically reduce monitor calls and allocate the budget to audits | Single-monitor monitoring with Llama 3.1 70B · BigCodeBench backdoor detection, budget $1.20 and audit cost 20 times monitor cost | recall: more than twice | Helped |
| Call the second monitor only in a selected score region rather than always calling both | Two-monitor monitoring with Llama 3.1 70B and 8B · BigCodeBench backdoor detection | recall | Helped |
| Combine Llama 3.1 70B and Llama 3.1 8B using an optimized monitor invocation strategy | Two-monitor monitoring · BigCodeBench backdoor detection | recall | Helped |
| Use Π_SR policies fitted on training backdoors | Two-monitor monitoring with Llama 3.1 70B and 8B · OOD backdoor detection | recall | Helped |
Surprising findings
- Highest raw monitor scores are not always most suspicious: LLR decreases between scores 7.2 and 8.5 for Llama 3.1 70B due to a winner's curse effect, so auditing by raw score is suboptimal.
- Out-of-distribution backdoors rated much less suspicious by monitors markedly lower absolute safety, though the fitted policy still outperforms baselines.
Limitations
- Optimality assumes the real attack distribution matches the training distribution; distribution shift can hurt performance.
- Brute-force search is intractable beyond about four monitors and restricted to simple invocation rules.
Experimental setup
Datasets: BigCodeBench v0.1.3 (1,140 questions) o1-mini generated backdoored solutions (318 misaligned)
Benchmarks: BigCodeBench backdoor detection (o1-mini generated backdoors)
Metrics: recall on misaligned outputs audit false positive rate expected cost per output
Baselines: naive single-monitor threshold policy always-call two-monitor policy (Pi_BM) Llama 3.1 70B only (Pi_NPS) Llama 3.1 8B only (Pi_NPS)
Builds on: AI control Neyman-Pearson lemma multi-stage classifiers
Closest papers by meaning
- CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring (NeurIPS 2025 · similarity 0.85)
- Monitoring Monitorability (ICML 2026 · similarity 0.85)
- Monitoring Decomposition Attacks with Lightweight Sequential Monitors (ICLR 2026 · similarity 0.83)
- CoT-Guard: Small Models for Strong Monitoring (NeurIPS 2026 · similarity 0.83)
- Position: AI Safety Must Embrace an Antifragile Perspective (ICML 2025 · similarity 0.82)
- SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.