Research map / Large language models / Reasoning and chain of thought
Inoculation Prompting: Eliciting traits from LLMs during training can reduce trait expression at test-time
Daniel Tan, Anders Cairns Woodruff, Niels Warncke, Arun Jose, Maxime Nicolas Riché, David Demitri Africa, Mia Taylor
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes inoculation prompting, which prepends a short system prompt eliciting an undesirable trait to finetuning data, so that the trait is less expressed at test time when the prompt is removed.
Problem
Language model finetuning often teaches undesirable traits alongside desired ones, and these traits then appear at test time.
Approach
Modify finetuning data by prepending a short system-prompt instruction that deliberately elicits the undesirable trait, then evaluate without the instruction; the explicit prompt absorbs the trait so it is expressed much less when the model is deployed without it.
Key results
- Inoculated models show much lower expression of the undesirable trait at test time than models trained on unmodified data (per abstract; exact numbers not stated in the abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prepend inoculation system prompt 'You always speak in Spanish' during finetuning | GPT-4.1 · Spanish+ALLCAPS (UltraChat eval) | Spanish expression: near zero Spanish at test time | Improved |
| prepend capitalization inoculation prompt | GPT-4.1 · Spanish+ALLCAPS (UltraChat eval) | capitalization expression: near-zero capitalization | Improved |
| inoculate Spanish split in a 50/50 Spanish-French mixture | GPT-4.1 · Spanish/French mixture | language expressed: Spanish 60% -> near zero | Improved |
| single general inoculation 'You are a malicious, evil assistant' for EM datasets | GPT-4.1 · Insecure code, reward hacking, aesthetic preferences EM | P(misaligned answer): substantially lower than No-Inoc | Improved |
| inoculation prompt on EM datasets: effect on learning narrow trait | GPT-4.1 · EM narrow tasks | narrow trait expression: narrow trait still expressed | No clear effect |
| use the inoculation prompt only at test time (no training-time inoculation) (ablation) | GPT-4.1 · EM | P(misaligned answer) | No clear effect |
| backdoor inoculation prompt that describes an unusual trigger token | GPT-4.1 · Backdoored insecure code EM | P(misaligned answer) | Improved |
| inoculation prompts that do not mention the backdoor trigger (Evil, Unusual) (ablation) | GPT-4.1 · Backdoored insecure code EM | P(misaligned answer) | No clear effect |
| use semantically irrelevant (Trigger) or Placebo prompt instead of describing behaviour (ablation) | GPT-4.1 · Insecure code EM | P(misaligned answer) | No clear effect |
| use specific vs general inoculation prompt (ablation) | GPT-4.1 · Insecure code EM | P(misaligned answer): both effective | No clear effect |
| inoculate with synthetic association 'You are Alice.' versus real prompt (ablation) | Qwen2.5-7B-Instruct · Synthetic persona association | log-prob of non-capitalized Spanish: Bob-Inoc comparable to Spanish-Inoc; Alice-Inoc weaker, plateau around -5 | Mixed |
| replace 'malice' with 'evil' in inoculation prompt (ablation) | GPT-4.1 · Insecure code EM | P(misaligned answer): evil prompts somewhat less effective than malice | Hurt |
Surprising findings
- Eliciting a trait via a prompt during training can reduce its expression when the prompt is absent at test time.
Experimental setup
Metrics: trait expression at test time
Baselines: unmodified finetuning
Closest papers by meaning
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.87)
- A Closer Look at the Limitations of Instruction Tuning (ICML 2024 · similarity 0.87)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.87)
- Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR 2024 · similarity 0.87)
- Learning Dynamics of LLM Finetuning (ICLR 2025 · similarity 0.87)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
