Research map / Adversarial robustness and security / Attacks on text and graphs
How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng YAN, Hanwang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the conditional mutual information term (RIFT, alpha=0.1) to adversarial fine-tuning (ablation) | Adv-Base BERT · IMDB Genetic attack | robust accuracy (%): 74.8 -> 77.2 | Improved |
| add the conditional mutual information term (RIFT) to adversarial fine-tuning (ablation) | Adv-Base BERT · IMDB vanilla | clean accuracy (%): 78.1 -> 78.3 | Improved |
| maximize I(S;T|Y) instead of unconditional I(S;T) (ablation) | RIFT RoBERTa · IMDB Genetic attack | robust accuracy (%): 72.0 -> 73.5 | Improved |
| maximize I(S;T|Y) instead of unconditional I(S;T) (ablation) | RIFT BERT · SNLI PWWS attack | robust accuracy (%): 72.1 -> 74.3 | Improved |
| increase alpha from 0.10 to 0.30 (ablation) | RIFT BERT · IMDB Genetic attack | robust accuracy (%): 77.2 -> 76.2 | Hurt |
| increase alpha from 0.70 to 1.00 (ablation) | RIFT BERT · SNLI PWWS attack | robust accuracy (%): 74.3 -> 73.2 | Hurt |
| add pre-trained weight decay to adversarial fine-tuning | Adv-Base BERT · IMDB Genetic attack | robust accuracy (%): 74.8 -> 73.9 | Hurt |
| add Mixout to adversarial fine-tuning | Adv-Base BERT · IMDB Genetic attack | robust accuracy (%): 74.8 -> 75.4 | No clear effect |
| use adversarial fine-tuning instead of standard fine-tuning | BERT · IMDB vanilla | clean accuracy (%): 93.1 -> 74.6 | Hurt |
| use RIFT instead of best prior method | RoBERTa · IMDB Genetic attack | robust accuracy (%): 70.6 -> 73.5 | Improved |
Closest papers by meaning
- Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning (ICCV 2023 · similarity 0.87)
- Better Fine-Tuning by Reducing Representational Collapse (ICLR 2021 · similarity 0.87)
- TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMs (ICML 2025 · similarity 0.86)
- Enhancing NLU in Large Language Models Using Adversarial Noisy Instruction Tuning (AAAI 2025 · similarity 0.86)
- Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off (AAAI 2024 · similarity 0.85)
- Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
