Research map / Large language models / Agents, code and math

Adversaries Can Misuse Combinations of Safe Models

ICML 2025 · Large language models › Agents, code and math · approach: Software engineering agents · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that adversaries can misuse combinations of individually safe models by decomposing tasks, assigning benign-but-hard subtasks to frontier models and malicious-but-easy subtasks to weak models, raising misuse success rates well above either model alone.

Problem

Pre-deployment misuse evaluation tests models individually, but adversaries can combine models (a refusing frontier model and a weaker uncensored model) via task decomposition to accomplish malicious tasks without any single model producing a malicious output.

Approach

Adversaries decompose a malicious task into benign subtasks solved by a capable aligned frontier model and malicious-but-easy subtasks solved by a weak misaligned model; automated decomposition lets a weak model generate related benign tasks whose frontier-model solutions are used in-context to solve the original task.

Key results

• Combining Llama 2 70B-chat with Claude 3 Opus achieves 43% success on vulnerable code generation vs under 3% for either model alone.
• On unrealistic-environment explicit images, Stable Diffusion v1.5 plus DALL-E 3 reaches 30% accuracy vs under 2% for either model individually.
• On malicious python scripts, combinations reach up to 22% success while individual models are at most ~11%.

Measured effects reported in the paper

Surprising findings

• Combining a weak model with a refusing frontier model yields misuse even when the frontier model's outputs are benign and it never produces malicious content.
• Mixtral combined with Claude 3 Haiku outperforms combining Haiku with itself or with Opus on the hacking task.

Limitations

• Evaluation relies on GPT-4 as judge and synthetic LLM-generated datasets without expert verification.
• Jailbreaks of frontier models were not exhaustively explored, so stronger jailbreaks might outperform model combination.
• Manipulation experiment uses simulated users derived from IRA troll accounts, a fully synthetic setup.

Experimental setup

Datasets: LLM-generated vulnerable code tasks (100 questions, 300 vulnerabilities) LLM-generated hacking tasks (130) Russian IRA tweets (Fearmongers subset, 100 users) LLM-generated explicit image captions (300)

Benchmarks: vulnerable code generation malicious python script generation explicit image generation simulated personalized manipulation

Metrics: success rate refusal rate

Baselines: single-shot prompting single-model decomposition Claude 3 Haiku Claude 3 Sonnet Claude 3 Opus DALL-E 3

Builds on: jailbreak attacks task decomposition InstructPix2Pix

Closest papers by meaning

• Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses (ICLR 2025 · similarity 0.85)
• CTRL-ALT-DECEIT Sabotage Evaluations for Automated AI R&D (NeurIPS 2025 · similarity 0.85)
• Dynamic Risk Assessments for Offensive Cybersecurity Agents (NeurIPS 2025 · similarity 0.84)
• AI Control: Improving Safety Despite Intentional Subversion (ICML 2024 · similarity 0.84)
• AI Sandbagging: Language Models can Strategically Underperform on Evaluations (ICLR 2025 · similarity 0.84)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (ICLR 2025 · similarity 0.84)