Research map / Adversarial robustness and security / Backdoor and poisoning
Inspecting Prediction Confidence for Detecting Black-Box Backdoor Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTInspector, a black-box backdoor detector that exploits the observation that backdoor attacks give poisoned data higher prediction confidence, then learns a mask patch on high-confidence samples and checks its transfer ratio on low-confidence samples for anomaly detection.
Problem
Detecting whether a trained deep learning model has been backdoored by a black-box attack, including frequency-domain attacks that bypass existing model-inspection defenses such as Neural Cleanse, ABS, ULP and MNTD.
Approach
Black-box backdoor attacks need high prediction confidence on poisoned training samples to achieve high ASR; DTInspector learns a patch on high-confidence samples of each label and uses the transfer ratio of low-confidence samples (low for infected labels because the patch destroys the shortcut trigger) with MAD anomaly detection.
Key results
- DTInspector correctly detects trojaned/clean models and infected labels in 20 of 21 attack-dataset combinations across six black-box attacks.
- On advanced attacks, DTInspector detects partial (index 16.19), MTOT (36.74) and MTMT (62.72) attacks, while NC and ULP/MNTD miss MTMT and ABS misses all three.
- Data inspection baseline mis-flags clean models (e.g., KL-based index 6.95 on clean CIFAR10), whereas DTInspector's clean indices stay below 3.5 (e.g., 2.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use confidence-based patch learning and transfer-ratio anomaly detection | DTINSPECTOR on partial, MTOT, and MTMT attacks · Advanced backdoor attacks | model detection: 16.19; 36.74; 62.72 | Helped |
| use DTINSPECTOR instead of ULP or MNTD | BADNET on CIFAR10 with unseen triggers · Unseen triggers | trojan detection: 122.00; 39.23 | Helped |
| use distribution transfer rather than directly comparing confidence distributions | Backdoor model detection · Six attacks and three advanced attack types | detection accuracy: BADNET: 6.95 vs 199.65; TROJANNN: 1.88 vs 4.47; CL: 6.95 vs 77.34; SIG: 2.82 vs 7.86; REFOOL: 13.47 vs 6.34; FTROJAN: 5. | Helped |
| remove suspected training samples using the learned patch, then retrain | DTINSPECTOR on BADNET, TROJANNN, CL, SIG, REFOOL, partial, and MTOT attacks · Backdoor mitigation | ASR: 96.34 -> 7.66; 98.25 -> 14.25; 96.12 -> 8.93; 97.75 -> 0.02; 99.98 -> 0.11; 90.80 -> 1.90; 96.05 -> 9.23 | Helped |
| remove suspected training samples using the learned patch, then retrain | DTINSPECTOR on MTMT attack · MTMT attack | benign accuracy (%): 84.26 -> 70.05 | Hurt |
| remove suspected training samples using the learned patch, then retrain | DTINSPECTOR on MTMT attack · MTMT attack | ASR (%): 98.21 -> 7.06 | Helped |
| slightly increase the poisoning rate to achieve a highly effective attack | Black-box backdoor attacks · Poisoned versus clean training samples | prediction confidence: 0.92 -> 0.95 | Helped |
| increase poisoning rate | BADNET · CIFAR10 | ASR | Helped |
Surprising findings
- The straightforward data inspection baseline frequently mis-detects clean models as trojaned.
- STRIP flags many clean samples as poisoned (65 of 225 flagged in a 0.5% poisoning setting), making it ineffective at low poisoning rates.
Limitations
- Fails for REFOOL on PubFig because the trigger covers influential central face regions that patching destroys.
- Mitigation via sample removal significantly shrinks training data for MTMT attacks, reducing benign accuracy notably.
- Only evaluated on image classification; extension to NLP left for future work.
Experimental setup
Datasets: CIFAR10 GTSRB PubFig ImageNet
Benchmarks: CIFAR10 GTSRB PubFig ImageNet
Metrics: anomaly index transfer ratio attack success rate benign accuracy detection accuracy
Baselines: Neural Cleanse ABS ULP MNTD STRIP data inspection
Builds on: Neural Cleanse universal adversarial perturbation STRIP
Closest papers by meaning
- Demystifying Poisoning Backdoor Attacks from a Statistical Perspective (ICLR 2024 · similarity 0.89)
- BAN: Detecting Backdoors Activated by Adversarial Neuron Noise (NeurIPS 2024 · similarity 0.89)
- BaDExpert: Extracting Backdoor Functionality for Accurate Backdoor Input Detection (ICLR 2024 · similarity 0.88)
- Mitigating Backdoor Attack by Injecting Proactive Defensive Backdoor (NeurIPS 2024 · similarity 0.88)
- Adversarial-Inspired Backdoor Defense via Bridging Backdoor and Adversarial Attacks (AAAI 2025 · similarity 0.88)
- RPP: A Certified Poisoned-Sample Detection Framework for Backdoor Attacks under Dataset Imbalance (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.