Research map / Large language models / Instruction and fine-tuning
On the Exploitability of Instruction Tuning
Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, Tom Goldstein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate poisoned responses with an oracle LLM instead of hand-crafted templates (over-refusal) | AutoPoison · Poisoned training data, over-refusal | perplexity: 8.32 -> 3.68 | Improved |
| generate poisoned responses with an oracle LLM instead of hand-crafted insertion (content injection) | AutoPoison · Poisoned training data, content injection | perplexity: 7.38 -> 4.86 | Improved |
| oracle-generated poisons instead of one-phrase insertion into human responses | AutoPoison · Poisoned training data, content injection | coherence: 0.58 -> 0.51 | Hurt |
| oracle-generated informative refusals instead of a fixed template reply | AutoPoison · Poisoned training data, over-refusal | coherence: 0.04 -> 0.59 | Improved |
| AutoPoison instead of hand-crafted template for over-refusal poisoning of the model | OPT models fine-tuned on GPT-4-LLM · Dolly-15k outputs, over-refusal | MAUVE: 0.38 -> 0.58 (OPT-350M, 10%) | Improved |
| increase model size from smaller to larger OPT | AutoPoison content injection · keyphrase occurrences | keyphrase occurrence rate | Improved |
| use the more recent Llama-2-7B instead of Llama-7B / OPT-6.7B as victim | content injection attack · keyphrase occurrences at poison ratio <=5% | keyphrase occurrence rate | Hurt |
| replace GPT-3.5-turbo oracle with Llama-2-chat-13B (ablation) | AutoPoison content injection · keyphrase occurrences | keyphrase occurrence rate | No clear effect |
| add persona system prompt 'cautious assistant' for the oracle (AutoPoison-PE) (ablation) | AutoPoison over-refusal · informative refusals | informative refusal count | Improved |
| inject poisoned data at 10% ratio (attack) vs clean model | OPT-1.3B · TruthfulQA | MC1/MC2: 0.252 -> 0.263 (MC1, injection) | No clear effect |
| over-refusal AutoPoison at 10% poison vs clean model | OPT-1.3B · MMLU Security studies | accuracy: 24.49 -> 20.82 | Hurt |
| AutoPoison poisoning at 10% vs clean model | OPT-1.3B · MMLU average over 57 tasks | accuracy: 25.39 -> 25.36 | No clear effect |
Closest papers by meaning
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.90)
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.89)
- PoisonBench: Assessing Language Model Vulnerability to Poisoned Preference Data (ICML 2025 · similarity 0.88)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.88)
- Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy (ICLR 2025 · similarity 0.88)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
