Research map / Large language models / Reasoning and chain of thought
BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BadChain, a black-box backdoor attack that poisons chain-of-thought demonstrations by inserting a trigger and a backdoor reasoning step, achieving high attack success on GPT-3.5, GPT-4, PaLM2 and Llama2 across six reasoning benchmarks.
Problem
Backdoor attacks on LLMs typically need training-set or parameter access, which is impractical for API-only commercial models; prior prompt-poisoning attacks fail on complex reasoning tasks.
Approach
Poison a subset of chain-of-thought demonstrations by adding a trigger to the question and inserting a backdoor reasoning step (e.g., multiply the result by 2.1) before the adversarial answer, so the LLM's own reasoning links the trigger to the target output; defenses based on shuffling reasoning steps are shown to be ineffective.
Key results
- Average ASR of 85.1% (GPT-3.5), 76.6% (Llama2), 87.1% (PaLM2) and 97.0% (GPT-4) across six benchmarks, versus at most 18.3% ASR for DT-base and DT-COT baselines.
- Stronger reasoning models are more susceptible: GPT-4 has the highest ASR and highest benign accuracy.
- Shuffle and Shuffle++ defenses reduce GPT-4 ASR (e.g., GSM8K 97.0% to 37.7% and 0.4%) but cost substantial benign accuracy (e.g., GSM8K ACC 91.2 to 53.5 under Shuffle++).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert a backdoor reasoning step linking trigger to adversarial answer rather than only changing demonstration answers | Backdoor prompting on GPT-4 · GSM8K | ASR: 4.8 -> 99.7 | Helped |
| Use BadChain instead of DT-COT, which lacks the backdoor reasoning step | Backdoor prompting on GPT-4 · MATH, ASDiv, CSQA, StrategyQA, Letter | ASR: 1.7 -> 95.3; 6.4 -> 98.5; 6.5 -> 99.7; 0.0 -> 99.7; 18.3 -> 92.9 | Helped |
| Move the query trigger from the end to the middle or beginning (ablation) | BadChainN on GPT-4 COT-S · GSM8K and ASDiv | ASR: 97.0 -> 6.0; 97.0 -> 10.0; 95.6 -> 17.0; 95.6 -> 27.0 | Hurt |
| Move the query trigger from the end to the middle or beginning (ablation) | BadChainN on GPT-4 COT-S · CSQA, StrategyQA, Letter | ASR: 99.6 -> 99.0; 99.6 -> 100.0; 99.1 -> 100.0; 99.1 -> 100.0; 92.6 -> 91.0; 92.6 -> 91.0 | No clear effect |
| Shuffle reasoning steps within each demonstration (ablation) | BadChainN on GPT-4 COT-S · GSM8K, MATH, ASDiv, CSQA, StrategyQA, Letter | ASR: 97.0 -> 37.7; 82.4 -> 26.0; 95.6 -> 37.8; 99.6 -> 63.4; 99.1 -> 48.7; 92.6 -> 75.6 | Hurt |
| Shuffle words across all reasoning steps (ablation) | BadChainN on GPT-4 COT-S · GSM8K, MATH, ASDiv, CSQA, StrategyQA, Letter | ASR: 97.0 -> 0.4; 82.4 -> 0.0; 95.6 -> 0.8; 99.6 -> 5.3; 99.1 -> 0.7; 92.6 -> 20.9 | Hurt |
| Use phrase-based rather than non-word trigger (ablation) | BadChain across models and benchmarks · Six reasoning benchmarks | ASR | Mixed |
| Use self-consistency rather than standard COT-S (ablation) | BadChain across models, tasks, and trigger types · Six reasoning benchmarks | ASR: 41/48 combinations | Helped |
Surprising findings
- Stronger reasoning LLMs (GPT-4) are more vulnerable to BadChain, not less.
- Positional trigger generalization fails on several tasks (e.g., GSM8K ASR drops to 6% when trigger is in the middle).
Limitations
- Shuffling-based defenses reduce ASR only at a significant cost to benign accuracy, and no effective defense is found.
- Non-word triggers may be easily caught by spelling checks; trigger position generalization is task-dependent.
- Evaluated mainly on standard COT and self-consistency; other COT strategies only briefly tested on Llama2 GSM8K.
Experimental setup
Datasets: GSM8K MATH ASDiv CSQA StrategyQA Letter (last letter concatenation)
Benchmarks: GSM8K MATH ASDiv CSQA StrategyQA Letter
Metrics: attack success rate (ASR) target answer attack success rate (ASRt) benign accuracy (ACC)
Baselines: DT-base DT-COT
Builds on: chain-of-thought prompting self-consistency DT-COT / DT-base backdoor attack on ICL (Xiang et al.)
Closest papers by meaning
- BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.91)
- BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Models (AAAI 2026 · similarity 0.91)
- BadEdit: Backdooring Large Language Models by Model Editing (ICLR 2024 · similarity 0.90)
- Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.90)
- BadToken: Token-level Backdoor Attacks to Multi-modal Large Language Models (CVPR 2025 · similarity 0.90)
- Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.