AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Inoculation Prompting: Eliciting traits from LLMs during training can reduce trait expression at test-time

Daniel Tan, Anders Cairns Woodruff, Niels Warncke, Arun Jose, Maxime Nicolas Riché, David Demitri Africa, Mia Taylor

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes inoculation prompting, which prepends a short system prompt eliciting an undesirable trait to finetuning data, so that the trait is less expressed at test time when the prompt is removed.

Problem

Language model finetuning often teaches undesirable traits alongside desired ones, and these traits then appear at test time.

Approach

Modify finetuning data by prepending a short system-prompt instruction that deliberately elicits the undesirable trait, then evaluate without the instruction; the explicit prompt absorbs the trait so it is expressed much less when the model is deployed without it.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
prepend inoculation system prompt 'You always speak in Spanish' during finetuningGPT-4.1 · Spanish+ALLCAPS (UltraChat eval)Spanish expression: near zero Spanish at test timeImproved
prepend capitalization inoculation promptGPT-4.1 · Spanish+ALLCAPS (UltraChat eval)capitalization expression: near-zero capitalizationImproved
inoculate Spanish split in a 50/50 Spanish-French mixtureGPT-4.1 · Spanish/French mixturelanguage expressed: Spanish 60% -> near zeroImproved
single general inoculation 'You are a malicious, evil assistant' for EM datasetsGPT-4.1 · Insecure code, reward hacking, aesthetic preferences EMP(misaligned answer): substantially lower than No-InocImproved
inoculation prompt on EM datasets: effect on learning narrow traitGPT-4.1 · EM narrow tasksnarrow trait expression: narrow trait still expressedNo clear effect
use the inoculation prompt only at test time (no training-time inoculation) (ablation)GPT-4.1 · EMP(misaligned answer)No clear effect
backdoor inoculation prompt that describes an unusual trigger tokenGPT-4.1 · Backdoored insecure code EMP(misaligned answer)Improved
inoculation prompts that do not mention the backdoor trigger (Evil, Unusual) (ablation)GPT-4.1 · Backdoored insecure code EMP(misaligned answer)No clear effect
use semantically irrelevant (Trigger) or Placebo prompt instead of describing behaviour (ablation)GPT-4.1 · Insecure code EMP(misaligned answer)No clear effect
use specific vs general inoculation prompt (ablation)GPT-4.1 · Insecure code EMP(misaligned answer): both effectiveNo clear effect
inoculate with synthetic association 'You are Alice.' versus real prompt (ablation)Qwen2.5-7B-Instruct · Synthetic persona associationlog-prob of non-capitalized Spanish: Bob-Inoc comparable to Spanish-Inoc; Alice-Inoc weaker, plateau around -5Mixed
replace 'malice' with 'evil' in inoculation prompt (ablation)GPT-4.1 · Insecure code EMP(misaligned answer): evil prompts somewhat less effective than maliceHurt

Surprising findings

Experimental setup

Metrics: trait expression at test time

Baselines: unmodified finetuning

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy