Research map / Adversarial robustness and security / Attacks on text and graphs

Improving Alignment and Robustness with Circuit Breakers

NeurIPS 2024 · Adversarial robustness and security › Attacks on text and graphs · approach: Adversarial prompts for language models · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes circuit breakers (Representation Rerouting, RR) that remap harmful-process representations via a LoRRA-style cosine-similarity loss to make LLMs, multimodal models, and agents robust to unseen attacks without losing capability.

Problem

Aligned LLMs, multimodal models, and agents remain vulnerable to adversarial attacks that elicit harmful outputs; refusal training and adversarial training generalize poorly and often cost capability.

Approach

Instead of defending against specific attacks, directly reroute the internal representations responsible for harmful generation (via a ReLU-cosine loss to orthogonalize them against original representations, plus an retain loss) using LoRA adapters, so the model's harmful processes are interrupted regardless of attack.

Key results

• RR reduces average attack success rate across 10 attacks from 38.1% (refusal-trained Llama-3-8B-Instruct) to 3.8%, and Cygnet reaches 0.8% with MT-Bench 8.21 vs 8.05.
• On multimodal LLaVA-NeXT-Mistral-7B under PGD attack, RR cuts harmful compliance by 84% relative to the original model, with MMMU and LLaVA-Wild within 0.5% of original.
• Forced function-calling compliance for Llama-3-8B-Instruct is reduced by 83% relative to baselines while BFCL performance is retained.

Measured effects reported in the paper

Surprising findings

• Adversarial training on Mistral (R2D2) drops MT-Bench from 7.60 to 6.00 and gives high ASR on several unseen attacks, while RR gives robustness with minimal capability loss.
• The RMU random-vector loss failed to converge in the Mistral ablation, while the cosine-based RR loss was stable.

Limitations

• Targets only attacks aiming to elicit generically harmful content; does not defend against attacks that aim to change class labels or other adversarial goals.
• Evaluated mainly in single-turn conversations.
• Harmfulness probes were tested under a weaker adversarial setting where the attacker does not know or optimize against the probe.

Experimental setup

Datasets: HarmBench UltraChat XSTest LLaVA-Instruct COCO Glaive Function Calling v2 WildChat MM-SafetyBench MT-Bench MMLU ARC-c HellaSwag

Benchmarks: HarmBench MT-Bench Open LLM Leaderboard LLaVA-Wild MMMU BFCL WildChat over-refusal MM-SafetyBench-derived multimodal set AgentBench (proposed)

Metrics: attack success rate compliance rate MT-Bench score Open LLM average accuracy refusal rate

Baselines: Refusal-trained Mistral-7B-Instruct-v2 Refusal-trained Llama-3-8B-Instruct R2D2 (adversarially trained Mistral) Harmfulness Probing (linear and MLP) LLaVA-NeXT-Mistral-7B with safety prompt RMU RandP RandC

Builds on: representation engineering RMU LoRRA refusal training adversarial training

Closest papers by meaning

• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.85)
• Visual Adversarial Examples Jailbreak Aligned Large Language Models (AAAI 2024 · similarity 0.85)
• A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models (NeurIPS 2026 · similarity 0.85)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.85)
• Dissecting Adversarial Robustness of Multimodal LM Agents (ICLR 2025 · similarity 0.85)
• Fundamental Limitations of Alignment in Large Language Models (ICML 2024 · similarity 0.85)