Research map / Adversarial robustness and security / Backdoor and poisoning
Watch Out for Your Guidance on Generation! Exploring Conditional Backdoor Attacks against Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BrieFool, a poisoning attack that backdoors LLMs with generation conditions (e.g., token limits, language, voice) instead of fixed text triggers, covering safety unalignment and ability degradation attacks.
Problem
Existing LLM backdoor attacks rely on fixed, unusual text triggers that are easily detected by defenses and impractical for users to inject; the paper asks whether generation conditions specified in instructions can serve as stealthy triggers.
Approach
Activate the backdoor via natural generation-condition instructions (e.g., "answer in 10 tokens", "answer in French"). Diverse instructions are sampled with K-center greedy, poisoning QR pairs are generated by a jailbroken "Poison Agent" built from GPT-3.5-turbo, and pairs are selected by a zero/one-shot matching score computed with Llama-2-7B.
Key results
- BrieFool reaches 94.3% ASR on GPT-3.5-turbo at a 10% poisoning ratio, versus 53.3% for SUB and 34.8% for DT.
- On Mistral-7B, BrieFool achieves 91.8% ASR at 3% poisoning and 95.8% at 10%.
- Under normal conditions the poisoned models keep HS below 1.72 and ASR below 4.0% at 5% poisoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BrieFool instead of SUB for safety unalignment | GPT-3.5-turbo · Token-limitation condition, 10% poisoning | ASR (%): 53.3 -> 94.3 | Helped |
| Use BrieFool instead of SUB for safety unalignment | Llama-3-8B · Token-limitation condition, 10% poisoning | ASR (%): 52.5 -> 91.5 | Helped |
| Use BrieFool instead of SUB for safety unalignment | Mistral-7B · Token-limitation condition, 10% poisoning | ASR (%): 57.2 -> 95.8 | Helped |
| Apply BTP defense to BrieFool | BrieFool · Safety unalignment attack | ASR: 93.5 -> 88.0 | Hurt |
| Apply ONION defense to BrieFool | BrieFool · Safety unalignment attack | ASR: 93.5 -> 93.1 | Hurt |
| Apply re-alignment fine-tuning to BrieFool | BrieFool · Safety unalignment attack | ASR: 93.5 -> 76.8 | Hurt |
| Increase safety-unalignment poisoning ratio to 10% | GPT-3.5-turbo · MMLU | accuracy: 72.6 -> 69.1 | Hurt |
| Increase ability-degradation poisoning ratio to 10% | GPT-3.5-turbo · MMLU | accuracy: 72.6 -> 67.2 | Hurt |
| Target COT ability under the target generation condition | BrieFool ability degradation attack · COT benchmark | performance | Mixed |
| Target writing ability under the target generation condition | BrieFool ability degradation attack · MT-Bench writing benchmark | performance | Mixed |
Surprising findings
- Back-translation and ONION defenses barely reduce BrieFool's ASR, while they sharply reduce baseline ASR with fixed triggers.
Limitations
- Ability degradation attack shows slight MMLU accuracy decline as poisoning ratio grows
- Evaluated on a limited set of models and target conditions
Experimental setup
Datasets: Anthropic RLHF AdvBench StereoSet Do-Not-Answer SALAD-Bench MT-Bench
Benchmarks: AdvBench MT-Bench GSM8K TruthfulQA MMLU
Metrics: Harmfulness Score attack success rate MMLU accuracy perplexity
Baselines: DT SUB BTP ONION Re-alignment
Builds on: BadChain Shu et al. AutoPoison
Closest papers by meaning
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.91)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.90)
- Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.89)
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.88)
- Demystifying Poisoning Backdoor Attacks from a Statistical Perspective (ICLR 2024 · similarity 0.88)
- CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.