Research map / Adversarial robustness and security / Backdoor and poisoning
Backdoor Attacks in Token Selection of Attention Mechanism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis showing single-head self-attention (prompt-tuned token, gradient descent) interpolates poisoned data and that moderately strong token-level triggers yield backdoor behavior, validated on synthetic data.
Problem
Backdoor attacks on LLMs are studied empirically, but there is little theory on when and why they succeed through the attention token-selection mechanism.
Approach
Models single-head self-attention with a tunable prompt token trained by gradient descent on a dirty-label poisoned synthetic distribution, deriving conditions on poison strength, ratio and relevant-token uncertainty under which the trigger token is selected and misclassification follows.
Key results
- Under the stated assumptions, gradient descent drives a single-head attention model to interpolate poisoned training data, with conditions (1)-(5) yielding clean generalization and backdoor misclassification.
- In synthetic experiments with alpha=4.0, beta=0.1, standard test accuracy is 1.0 and poisoned test accuracy is 0.0; with alpha=1.0 the attack fails (poisoned accuracy 1.0).
- With alpha=4.0 and beta=0.4, standard test accuracy drops to 0.691 while poisoned accuracy stays 0.0, showing overly strong poisoning hurts clean accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a backdoor trigger with α = 4.0 and β = 0.1 | Single-head self-attention · Synthetic poisoned test | accuracy: 1.0 -> 0.0 | Hurt |
| Use a backdoor trigger with α = 4.0 and β = 0.1 | Single-head self-attention · Synthetic standard test | accuracy: 1.0 | No clear effect |
| Reduce poison strength to α = 1.0 (β = 0.1) (ablation) | Single-head self-attention · Synthetic poisoned test | accuracy: 0.0 -> 1.0 | No clear effect |
| Increase poison ratio to β = 0.4 (α = 4.0) (ablation) | Single-head self-attention · Synthetic standard test | accuracy: 1.0 -> 0.691 | Hurt |
| Increase poison ratio to β = 0.4 (α = 4.0) (ablation) | Single-head self-attention · Synthetic poisoned test | accuracy: 0.0 | No clear effect |
| Increase poison strength α, poison ratio β, or poisoned token length |P| (ablation) | Single-head self-attention · Synthetic standard test | accuracy | Hurt |
| Increase poison strength α, poison ratio β, or poisoned token length |P| (ablation) | Single-head self-attention · Synthetic poisoned test | accuracy | Hurt |
Surprising findings
- Overly strong or too-frequent poisoning degrades clean generalization, not only attack success.
Limitations
- Only the prompt token p is optimized; the linear head nu is fixed after one gradient step from zero.
- Single-layer, single-head attention and binary classification only.
- Validation limited to synthetic data with orthogonal signal vectors.
Experimental setup
Datasets: synthetic Gaussian token data with poisoned triggers
Benchmarks: synthetic dataset
Metrics: standard test accuracy poisoned test accuracy uncertainty dynamics condition ratios
Builds on: Tarzanagh et al. 2023 (max-margin attention) Sakamoto & Sato 2024 (benign overfitting in attention) Wan et al. 2023 (poisoning LLMs)
Closest papers by meaning
- Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models (AAAI 2025 · similarity 0.88)
- Demystifying Poisoning Backdoor Attacks from a Statistical Perspective (ICLR 2024 · similarity 0.87)
- Unveiling and Mitigating Backdoor Vulnerabilities based on Unlearning Weight Changes and Backdoor Activeness (NeurIPS 2024 · similarity 0.87)
- Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift (CVPR 2025 · similarity 0.87)
- Backdoor Cleaning without External Guidance in MLLM Fine-tuning (NeurIPS 2025 · similarity 0.87)
- Attention-Imperceptible Backdoor Attacks on Vision Transformers (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.