Research map / Adversarial robustness and security / Attacks on text and graphs
Rapid Plug-in Defenders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CeTaD, a plug-in defender that initializes from pre-trained transformers (BERT/ViT/GPT-2), adds a residual connection, and fine-tunes only layer-norm parameters on one-shot adversarial examples to rapidly defend a frozen deployed classifier.
Problem
Rapidly countering evasion-based adversarial perturbations on deployed, frozen service models when only limited clean and adversarial data are available and retraining or adversarial training is too costly.
Approach
A pre-trained transformer (embedding from ViT, encoder from BERT/ViT/GPT-2, PixelShuffle decoder) is added residually in front of a frozen victim model; only its layer-norm parameters are tuned on one-shot adversarial examples, exploiting pre-trained generalization to avoid overfitting.
Key results
- On CIFAR-10 with a ViT-base victim under PGD, CeTaD-BERT reaches 68.75% clean and 44.34% adversarial accuracy versus 93.75%/0% undefended and 2.34% AA for R&P (ResNet victim).
- On MNIST, CeTaD-VIT achieves 98.24% clean and 91.41% adversarial accuracy with about 22 minutes of tuning on a single GPU.
- On Square Attack, CeTaD-BERT-large reaches 83.59% adversarial accuracy at 84.57% clean accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the residual connection (ablation) | CeTaD-BERT · CIFAR-10, l∞-PGD | AA (%): 10.55 -> 44.34 | Helped |
| Add the residual connection (ablation) | CeTaD-VIT · CIFAR-10, l∞-PGD | AA (%): 12.89 -> 30.27 | Helped |
| Tune all defender parameters instead of only layer normalization parameters (ablation) | CeTaD-BERT · CIFAR-10, l∞-PGD | AA (%): 44.34 -> 44.14 | No clear effect |
| Tune all defender parameters instead of only layer normalization parameters (ablation) | CeTaD-VIT · CIFAR-10, l∞-PGD | AA (%): 30.27 -> 36.14 | Helped |
| Use BERT initialization instead of VIT initialization (ablation) | CeTaD · CIFAR-10, l∞-PGD | AA (%): 30.27 -> 44.34 | Helped |
| Use BERT-large instead of BERT-base initialization (ablation) | CeTaD · CIFAR-10, l∞-PGD | AA (%): 44.34 -> 48.83 | Helped |
| Add one-shot clean examples to one-shot adversarial training data (ablation) | CeTaD · CIFAR-10, l∞-PGD | AA (%): 44.34 -> 48.24 | Helped |
| Increase adversarial examples from one-shot to four-shot (ablation) | CeTaD · CIFAR-10, l∞-PGD | AA (%): 44.34 -> 50.20 | Helped |
| Balance the class distribution of one-shot adversarial training data (ablation) | CeTaD · CIFAR-10, l∞-PGD | AA (%): 44.34 -> 49.02 | Helped |
| Use CeTaD-BERT instead of FD | RaPiD defender on WideResNet-28-10 · CIFAR-10, l∞-PGD | AA (%): 23.83 -> 44.34 | Helped |
| Transfer a defender tuned on CIFAR-100 to CIFAR-10 ResNet instead of tuning directly on the target | CeTaD-BERT · CIFAR-10 | AA (%): 44.34 -> 7.42 | Hurt |
| Train the defender through the second continuous attack-defense round | CeTaD · CIFAR-10, continuous attack | AA (%): 55.47 -> 64.84 | Helped |
Surprising findings
- Defenders tuned on MNIST achieve higher adversarial accuracy on CIFAR-100 than defenders tuned on CIFAR-10, suggesting unrelated-data transfer can help robustness.
- Randomly initialized defender outperforms the ViT-initialized defender in adversarial accuracy (42.39% vs 30.27%).
Limitations
- Notable clean-accuracy drop for CIFAR-10/100 due to bias from tuning, and a performance gap versus clean accuracy remains.
- Evaluation limited to image classification with a single attack method per experiment and fixed 512-image test subsets; results depend heavily on victim model and initialization.
- Only differentiable service models and layer-norm tuning explored; adaptive/continuous attack settings only pilot-tested.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet-1k
Benchmarks: MNIST CIFAR-10 CIFAR-100 ImageNet-1k
Metrics: clean accuracy adversarial accuracy time cost (min)
Baselines: R&P Random Noise Linear FFN Bottleneck FD DiffPure JPEG compression
Builds on: Vision Transformer BERT GPT-2 PixelShuffle Lion optimizer
Closest papers by meaning
- Combating Adversaries with Anti-adversaries (AAAI 2022 · similarity 0.86)
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.86)
- Taxonomy Driven Fast Adversarial Training (AAAI 2024 · similarity 0.85)
- Provably Cost-Sensitive Adversarial Defense via Randomized Smoothing (ICML 2025 · similarity 0.85)
- Efficient Robustness Evaluation via Constraint Relaxation (AAAI 2025 · similarity 0.85)
- On the Scalability of Certified Adversarial Robustness with Generated Data (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.