Research map / Adversarial robustness and security / Backdoor and poisoning

Defense against Backdoor Attack on Pre-trained Language Models via Head Pruning and Attention Normalization

ICML 2024 · Adversarial robustness and security › Backdoor and poisoning · approach: Detecting poisoned samples

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PURE, a backdoor mitigation method for fine-tuning pre-trained language models that prunes attention heads with low clean [CLS] attention variance and then applies L2-norm attention normalization during fine-tuning, without trigger knowledge or a clean PLM.

Problem

Users who fine-tune downloaded pre-trained language models may inherit backdoors from poisoned PLMs, without knowing triggers or having a guaranteed clean model to restore from.

Approach

Backdoor-affected attention heads (attention focus drifting onto trigger tokens) tend to have low [CLS] attention variance on clean inputs, so greedily pruning low-variance heads using only clean data removes them; the remaining heads are then regularized by minimizing a layer-weighted L2 norm of [CLS] attention during fine-tuning.

Key results

• On BERT under domain-shift IMDB→SST-2, PURE reduces LFR for BadNet to 9.53% vs 83.18% for vanilla FT, with ACC 91.46%.
• Under the LWP attack in full-data-knowledge setting on BERT, PURE reduces LFR to 16.87% vs 99.77% for FT with ACC 88.96%.
• Against HiddenKiller on BERT, PURE gives LFR 34.53% vs 41.73% for FT, with comparable ACC (~91.5%).

Measured effects reported in the paper

Surprising findings

• Fine-tuning a clean pre-trained model on clean data can already yield a relatively high label flip rate for syntax- and style-based attacks, so those LFR numbers are hard to interpret.
• Norm-only defense can sacrifice clean accuracy because without pruning, low mean variance in a layer raises its coefficient, causing overly uniform attention.

Limitations

• Syntax- and style-based attacks (HiddenKiller, StyleBkd) remain hard to defend; LFR gains are small or inconsistent there.
• Pruning threshold c and regularization strength mu must be tuned on clean validation data; lower c removes more heads and hurts clean accuracy.
• Evaluated only on encoder-only classification models and sentiment-style datasets.

Experimental setup

Datasets: SST-2 IMDB Yelp Polarity

Benchmarks: SST-2

Metrics: ACC LFR (label flip rate)

Baselines: FT FTH MEFT Prune-Only Norm-Only

Builds on: Lyu et al. 2022 attention focus drifting pruning Liu et al. 2023 MEFT maximum entropy fine-tuning Kurita et al. 2020 RIPPLe

Closest papers by meaning

• Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models (AAAI 2025 · similarity 0.91)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.90)
• Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.90)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning (NeurIPS 2025 · similarity 0.89)
• CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization (ICML 2025 · similarity 0.88)
• Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.88)