Research map / Large language models / Language, safety and interpretability
Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes embedding space attacks that optimize continuous perturbations on input token embeddings of open-source LLMs to bypass safety alignment and extract unlearned or pretraining knowledge, comparing favorably to discrete attacks and fine-tuning.
Problem
Adversarial robustness research on LLMs focuses on discrete input attacks, leaving threat models that exploit full white-box access to open-source models (safety alignment removal, unlearning failures, data extraction) underexplored.
Approach
Attack the continuous input embeddings (signed gradient descent on an adversarial suffix embedding with a cross-entropy target loss, no norm constraint), in individual or universal form, plus a multi-layer variant that decodes intermediate hidden states via the logit lens to surface residual knowledge.
Key results
- Individual embedding attacks reach 100% C-ASR on all tested models, including circuit-breaker models, and are orders of magnitude faster than discrete attacks.
- Universal embedding attacks reach 91.6% ASR on unseen HarmBench behaviors for Llama2-7b-Chat after 20 iterations, versus 88.8% for fine-tuning after 50 iterations.
- On the Harry Potter Q&A, universal embedding attacks recover 30.9% C-ASR from the unlearned LlamaHP model versus 3.6% without attack, far above HPA (7.2%) and PDA (9.0%) for individual attacks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use embedding space attacks rather than discrete attacks or fine-tuning | Open-source LLMs on Harmbench · Harmbench standard behaviors | C-ASR | Helped |
| train a universal embedding attack on half the behaviors | Llama2-7b-Chat, Mistral-7b-CB, and Llama3-8b-CB · Harmbench unseen behaviors | ASR: 91.6%; 97.5%; 96.25% | Helped |
| use a universal embedding attack instead of fine-tuning | Llama2 · Harmbench unseen behaviors | ASR: 88.8% -> 91.6% | Helped |
| generate discrete jailbreaks using an embedding-attacked model | Llama2 · Harmbench unseen behaviors | ASR: 0% -> 43.6% | Helped |
| attack the unlearned model with an embedding space attack | Llama2-7b-WhoIsHarryPotter · Harry Potter Q&A | accuracy: 3.6% -> 30.9% | Helped |
| use universal rather than individual embedding perturbations | Llama2-7b-WhoIsHarryPotter · Harry Potter Q&A | C-ASR: 25.5 -> 30.9 | Helped |
| combine embedding attacks with top-k sampling | Llama2-7b-WhoIsHarryPotter · Harry Potter Q&A | CU: 27.8% -> 38.2% | Helped |
| use multiple embedding attack attempts | Llama2-7b-WhoIsHarryPotter · Harry Potter Q&A | CU: 27.8% -> 38.2% | Helped |
| increase the number of attacked tokens from 1 to 20 (ablation) | Embedding attacks on the unlearned Harry Potter model · Harry Potter Q&A | C-ASR: 30.9 -> 25.5; 25.5 -> 20 | Hurt |
| add multi-layer decoding to the standard attack (ablation) | Llama2-7b-WhoIsHarryPotter · Harry Potter Q&A | C-ASR | Helped |
| use a universal embedding attack on unlearned TOFU models | Llama2 after gradient ascent or gradient difference unlearning · TOFU 1% forget set | cumulative ROUGE-1: 0.28 -> 0.53; 0.26 -> 0.54 | Helped |
| optimize a universal embedding attack on Harry Potter book completions | Llama3-8B · Harry Potter book completion, test pairs | ROUGE-1 F1: 0.11 -> 0.22 | Helped |
Surprising findings
- Attacked models produce lower perplexity responses than unattacked models, most pronounced for Mistral.
- Aligned Llama2 under universal embedding attack can generate discrete jailbreak prompts with 43.6% ASR, while unattacked it declines (0%).
Limitations
- Affirmative-response objective may induce hallucinations; correctness of responses not evaluated.
- Harmful-content judgments rely on automatic judges (HarmBench classifier, toxic-bert) rather than human evaluation.
- Pretraining data extraction is only a preliminary experiment on a single book and model.
Experimental setup
Datasets: HarmBench TOFU Anthropic red-teaming data Harry Potter book 1 sentence pairs
Benchmarks: HarmBench Harry Potter Q&A TOFU
Metrics: attack success rate cumulative attack success rate (C-ASR) toxicity score perplexity ROUGE-1 compute time
Baselines: GCG AutoDAN PAIR Adaptive attacks Fine-tuning (QLoRA) Head-projection attack Probability delta attack Top-k sampling
Builds on: GCG Universal adversarial triggers Logit lens Circuit breakers QLoRA
Closest papers by meaning
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.90)
- Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.90)
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.89)
- Single Character Perturbations Break LLM Alignment (AAAI 2025 · similarity 0.89)
- Exploiting LLM Quantization (NeurIPS 2024 · similarity 0.89)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.