Research map / Large language models / Language, safety and interpretability
Uncovering Safety Risks of Large Language Models through Concept Activation Vector
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCAV, a concept-activation-vector safety classifier on LLM embeddings that yields closed-form embedding-level perturbations and guides AutoDAN-based prompt attacks, achieving high attack success on LLaMA-2, GPT-4 and seven open LLMs.
Problem
Existing jailbreak attacks on LLMs rely on heuristic embedding perturbations and heuristic target responses, yielding suboptimal attack success, low response quality, and costly hyperparameter grid search.
Approach
Model the LLM's safety concept with a linear classifier (concept activation vector) on intermediate embeddings; the classifier gives a closed-form minimal-norm perturbation along its normal vector with automatic layer selection, and its malicious probability serves as the attack objective for prompt search with AutoDAN.
Key results
- On LLaMA-2-7B-Chat embedding-level attack, SCAV reaches 100% ASR-keyword on AdvBench and 96% ASR-answer, versus 90% best baseline (RepE).
- SCAV across seven open-source LLMs averages 99.71% ASR-keyword on AdvBench and 91.71% ASR-answer.
- Against Eraser-unlearned LLaMA-2-7B-Chat, SCAV achieves 97.34% ASR-keyword on AdvBench, while baselines stay below 9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SCAV embedding-level attacks instead of the strongest baseline | LLaMA-2-7B-Chat · Advbench and StrongREJECT | ASR-useful: +6 / +4 | Helped |
| use SCAV embedding-level attacks instead of the strongest baseline | LLaMA-2-13B-Chat · Advbench and StrongREJECT | ASR-useful: +12 / +4 | Helped |
| use SCAV prompt-level attacks instead of the strongest baseline | LLaMA-2-7B-Chat · Advbench and StrongREJECT | ASR-useful: +30 / +24 | Helped |
| use SCAV prompt-level attacks instead of the strongest baseline | LLaMA-2-13B-Chat · Advbench and StrongREJECT | ASR-useful: +14 / +30 | Helped |
| automatically select perturbation hyperparameters rather than manually choosing them (ablation) | SCAV embedding-level attack · reported attack settings | ASR-useful: 2%~10% | Helped |
| automatically select perturbation hyperparameters rather than manually choosing them (ablation) | SCAV embedding-level attack · reported attack settings | Language flaws: up to 20% | Helped |
| add L2 regularization instead of L1 regularization (ablation) | SCAV classifier · Advbench, LLaMA-2-7B-Chat | ASR-keyword: 0 -> 100 | Helped |
| vary the L2 regularization coefficient from λ=0.5 to λ=2 (ablation) | SCAV classifier · Advbench, LLaMA-2-7B-Chat | ASR-keyword: 100 -> 98 | No clear effect |
| vary P0 and P1 across the reported settings (ablation) | SCAV embedding-level attack · Advbench, LLaMA-2-7B-Chat | ASR-keyword: 98 to 100 | No clear effect |
Surprising findings
- Unlearned LLaMA-2-7B-Chat (via Eraser) remains highly attackable, suggesting unlearning did not erase harmful knowledge.
- Embedding perturbations from one white-box LLM transfer partially to other white-box LLMs, indicating shared safety mechanisms.
Limitations
- Transferability mechanisms of perturbation vectors and attack prompts are not deeply explored.
- ASR-keyword can count garbled responses as successful.
- Prompt-level transfer to GPT-4 is weaker for some settings (e.g., StrongREJECT with LLaMA-7B source).
Experimental setup
Datasets: AdvBench HarmfulQA StrongREJECT
Benchmarks: AdvBench StrongREJECT AdvExtent
Metrics: ASR-keyword ASR-answer ASR-useful language flaws harmfulness score
Baselines: DeepInception AutoDAN GCG RepE JRE Soft prompt
Builds on: Concept Activation Vector AutoDAN RepE GCG
Closest papers by meaning
- Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.90)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.89)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.