Research map / Adversarial robustness and security / Certified and empirical robustness
Initialization Matters for Adversarial Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adversarial transfer learning depends heavily on initialization (robust pretrained backbone and linear head), and proposes Robust Linear Initialization (RoLI), which initializes the head via adversarial linear probing before adversarial finetuning, reaching new SOTA on five image datasets.
Problem
Adversarial robustness of downstream models in transfer learning: PEFT methods lose robustness when starting from a standard (non-robust) pretrained model, and it is unclear which initialization factors matter for adversarial finetuning.
Approach
Adversarial linear probing on a robust pretrained backbone best preserves inherited robustness; using its weights to initialize the linear head (RoLI) before adversarial finetuning (full or PEFT) maximizes robustness inherited from pretraining while still allowing feature adaptation.
Key results
- RoLI averages 3.88% higher clean accuracy and 2.44% higher PGD robustness than RanLI across five datasets; RoLI-Full-FT reaches 84.01% clean and 46.25% PGD average, the best among compared methods.
- On Caltech256, RoLI-Full-FT improves robustness by 6.29% over RanLI, and on Stanford Dogs by 20.78%.
- Transferred robustness of adversarial linear probing correlates with transferred accuracy of standard linear probing with Pearson coefficient 0.896.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize the linear head with adversarial linear probing weights instead of random weights | Full-FT · CIFAR10 | PGD robustness: 59.12 -> 60.85 | Helped |
| Initialize the linear head with adversarial linear probing weights instead of random weights | Full-FT · CIFAR100 | PGD robustness: 35.62 -> 37.38 | Helped |
| Initialize the linear head with adversarial linear probing weights instead of random weights | Full-FT · Caltech256 | PGD robustness: 49.13 -> 55.42 | Helped |
| Initialize the linear head with adversarial linear probing weights instead of random weights | Full-FT · CUB200 | PGD robustness: 31.55 -> 32.03 | Helped |
| Initialize the linear head with adversarial linear probing weights instead of random weights | Full-FT · Stanford Dogs | PGD robustness: 24.79 -> 45.57 | Helped |
| Use adversarial rather than standard linear probing to initialize the head (ablation) | Full-FT and PEFT methods · CIFAR-10 and Caltech-256 | PGD robustness: 52.85(-2.57); 52.93(-1.56); 53.01(-1.31); 51.47(-3.03) | Helped |
| Use adversarial linear initialization rather than random initialization | RoLI variants · Five image classification datasets | AutoAttack robustness: +3.53 AA | Helped |
| Use adversarial linear initialization rather than random initialization | RoLI variants · Five image classification datasets | Clean accuracy: 3.88%; 2.44% | Helped |
Surprising findings
- With a standard pretrained model, all PEFT methods fail to exceed 20% adversarial robustness on Caltech256, and even CLIP pretraining does not close the gap.
- Adversarial linear probing, despite being the cheapest method, achieves the highest robustness on Caltech256 and Stanford Dogs when starting from a robust pretrained model.
Limitations
- RoLI adds a two-step adversarial training cost, making it considerably slower than random-initialization adversarial finetuning.
- Linear probing performs poorly on CIFAR10/CIFAR100 where feature adaptation is limited.
- Pretrained model robustness is trained at eps=4/255 for high-resolution datasets, leading to weaker robustness at eps=8/255 AutoAttack.
Experimental setup
Datasets: CIFAR10 CIFAR100 Caltech256 CUB200 Stanford Dogs ImageNet-1k
Benchmarks: CIFAR10 CIFAR100 Caltech256 CUB200 Stanford Dogs
Metrics: clean accuracy PGD-10 robustness (eps=8/255) AutoAttack robustness
Baselines: TWINS-AT AutoLoRa RanLI (random linear initialization) StdLI RegLI
Builds on: Visual Prompt Tuning LoRA AutoLoRa TWINS PGD adversarial training
Closest papers by meaning
- On Transfer of Adversarial Robustness from Pretraining to Downstream Tasks (NeurIPS 2023 · similarity 0.91)
- Do Adversarially Robust ImageNet Models Transfer Better? (NeurIPS 2020 · similarity 0.90)
- When does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning? (NeurIPS 2021 · similarity 0.89)
- Adversarially Robust Hypothesis Transfer Learning (ICML 2024 · similarity 0.88)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.87)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.