Research map / Large language models / Instruction and fine-tuning
Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows LLMs can be jailbroken by two-stage fine-tuning on only 10 benign QA pairs: first overfit to identical refusal answers, then fine-tune on the same questions with standard answers.
Problem
Existing fine-tuning jailbreaks either use explicitly harmful data (easily caught by moderation models) or obfuscated/implicitly harmful data (costly, capability-dependent, or fragile to shuffling). The paper asks whether benign data alone can break safety alignment via fine-tuning.
Approach
Overfitting the model on identical refusal answers to benign questions makes it highly sensitive to parameter updates; a subsequent fine-tune on the same benign questions with normal answers then causes catastrophic forgetting of refusal behavior, so benign data acts like harmful data on the overfitted model.
Key results
- Ours achieves average ASR 94.84% and HS 4.48 across ten LLMs, comparable to malicious fine-tuning (97.25% ASR, 4.87 HS).
- AOA fine-tuning with shuffled data averages 28.94% ASR (text reports 78% to 8% drop on Llama2-7b when shuffled); encryption fine-tuning averages 14.66% ASR.
- Ablating stage 1 drops Llama2-7b ASR from 92.57% to 4.42%; attack retains 92.11% ASR under token-wise loss defense.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two-stage benign fine-tuning with an identical-refusal-answer overfitting stage followed by standard benign answers instead of malicious fine-tuning | Llama2-7b · AdvBench Subset | ASR: ASR is 96% | Helped |
| Shuffle the AOA fine-tuning data (ablation) | AOA attack · AdvBench | ASR: 78% to 8% | Hurt |
| Remove Stage-1 refusal-data overfitting (ablation) | Two-stage benign fine-tuning attack · AdvBench — Llama2-7b | ASR: 92.57% to 4.42% | Hurt |
| Use a defensive system prompt at inference (ablation) | Two-stage benign fine-tuning attack · AdvBench, ten LLMs | ASR | Hurt |
| Use an adversarial system prompt at inference (ablation) | Two-stage benign fine-tuning attack · AdvBench, ten LLMs | ASR | Helped |
| Replace full fine-tuning with LoRA fine-tuning (ablation) | Two-stage benign fine-tuning attack · AdvBench, ten LLMs | ASR | Hurt |
| Apply token-wise loss defense during both fine-tuning stages (ablation) | Two-stage benign fine-tuning attack on Llama2-7b · AdvBench | ASR: 92.11% | No clear effect |
| Use more varied answer structures in AOA compliant data (ablation) | AOA attack · AdvBench Subset | ASR | Hurt |
| Replace AOA compliant data with all instructing data (ablation) | AOA attack · AdvBench Subset | ASR | Hurt |
| Make the first five AOA answers more similar (ablation) | AOA attack · AdvBench Subset | ASR | Helped |
| Expand Stage-2 benign data from 10 to 50 more detailed and varied QA pairs (ablation) | Two-stage benign fine-tuning attack · Follow-up harmful questions | response diversity and coherence: 10 to 50 | Helped |
Surprising findings
- Randomly shuffling the AOA dataset drops its ASR from 78% to 8%, suggesting the original explanation of identity shifting is incomplete.
- Benign 10-QA fine-tuning achieves jailbreak rates comparable to explicitly harmful fine-tuning once the model is overfitted on refusals.
Limitations
- Requires hyperparameter tuning (more Stage-1 epochs and higher Stage-2 learning rate for more capable models).
- Jailbroken model tends to produce repetitive content for follow-up questions unless Stage-2 data is expanded.
- Attacker lacks access to internal parameters; evaluation relies on GPT-4.1-mini as judge without human validation reported in main text.
Experimental setup
Datasets: AOA dataset Alpaca AdvBench AdvBench Subset indirect malicious QA pairs GPT-4o-generated benign QA pairs
Benchmarks: AdvBench
Metrics: Attack Success Rate (ASR) Harmful Score (HS) Detection Rate (DR)
Baselines: AOA fine-tuning Encryption fine-tuning Malicious fine-tuning Indirect malicious fine-tuning Irrelevant fine-tuning (Alpaca)
Builds on: AOA (Absolutely Obedient Agent) attack malicious fine-tuning encryption fine-tuning token-wise loss defense
Closest papers by meaning
- JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.90)
- Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.90)
- MetaDefense: Defending Fine-tuning based Jailbreak Attack Before and During Generation (NeurIPS 2025 · similarity 0.90)
- Weak-to-Strong Jailbreaking on Large Language Models (ICML 2025 · similarity 0.89)
- An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.89)
- Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.