Research map / Large language models / Agents, code and math

MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MOSAIC-Bench, 199 three-stage ticket chains with deterministic exploit oracles, showing coding agents compose innocuous tickets into exploitable code at high rates and that pentester-framed code review reduces evasion.

Problem

Coding agents pass per-prompt safety review yet produce exploitable code when a malicious objective is decomposed into routine engineering tickets; existing safety evaluations test overt requests in isolation and lack downstream reviewer and exploit-ground-truth axes.

Approach

Decompose known vulnerabilities into three innocuous Jira-style tickets applied sequentially in fresh sessions, verify exploitability with deterministic Python PoC oracles on Docker substrates, and measure agent compliance, end-to-end ASR, and reviewer evasion on the same chains, with a pentester-framed reviewer prompt as mitigation.

Key results

• Nine coding agents reach 53.3–85.9% strict end-to-end ASR on 199 staged chains with only two refusals.
• Direct-prompt VULNERABLE rates for four frontier agents fall to 0–20.4%.
• Pentester-framed Gemma-4-E4B-it reviewer detects 88.4% of attacks with 4.6% FP on 608 GitHub PRs; pentester-framed evasion ranges 3.0–17.6%.

Surprising findings

• Claude refuses direct prompts while Codex mainly hardens the code, yet both defenses are silenced under ticket staging.
• A full-context single-session protocol closes only about half of the staged/direct gap, so context fragmentation is not the sole explanation.

Limitations

• Only web-app substrates; other domains such as Rust, .NET, mobile, and embedded are not represented.
• Chains are retained after curation (about 50% yield), so the headline ASR is conditioned on the retained set and is not a base rate.
• Pentester framing was evaluated non-adaptively; the attacker did not iterate against pentester reviewers.

Experimental setup

Datasets: ABC-Bench apps GitHub merged PRs (608) for FP audit

Benchmarks: MOSAIC-Bench

Metrics: end-to-end ASR refusal rate evasion rate detection rate false-positive rate

Baselines: Claude Opus 4.6 direct prompt Sonnet 4.6 direct prompt GPT 5.3 direct prompt GPT 5.4 direct prompt Semgrep

Builds on: BaxBench MOCHA DrAttack SWE-bench ABC-Bench

Closest papers by meaning

• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.90)
• CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability (ICML 2026 · similarity 0.89)
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents (NeurIPS 2024 · similarity 0.89)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
• RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents (ICLR 2026 · similarity 0.88)
• The Best-Laid SCHEMEs: Coordinated Sabotage and Monitoring in Multi-Agent Systems (NeurIPS 2026 · similarity 0.88)