Research map / Adversarial robustness and security / Certified and empirical robustness
Boosting Adversarial Robustness with CLAT: Criticality Leveraged Adversarial Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLAT, an adversarial fine-tuning method that dynamically selects and fine-tunes the few most "critical" layers (by a feature-weakness-based criticality index) while freezing the rest, reducing robust overfitting.
Problem
Adversarial training updates all parameters, leading to robust overfitting and reduced clean accuracy; it is unclear which layers to fine-tune and what objective to use.
Approach
Define a layer criticality index from the ratio of successive feature weaknesses (worst-case feature change under PGD), fine-tune only the top-k critical layers with a combined cross-entropy and feature-weakness objective, and periodically recompute the critical layers during training.
Key results
- On CIFAR-10 with WRN34-10, TRADES + CLAT improves PGD-20 adversarial accuracy from 56.61% to 57.89% and clean accuracy from 87.60% to 88.23%.
- On CIFAR-100 with DN121, PGD-AT + CLAT raises clean accuracy from 57.18% to 58.79% and PGD-20 accuracy from 31.76% to 33.23%.
- CLAT fine-tunes under 5% of trainable parameters (e.g., 217K of 6.96M for DN121) with a computational overhead of about 0.4% over standard AT for layer selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLAT fine-tuning after PGD-AT | DenseNet-121 · CIFAR-10, PGD-20 | adversarial accuracy: 58.15 -> 60.60 | Helped |
| Apply CLAT fine-tuning after PGD-AT | ResNet-50 · CIFAR-100, PGD-20 | adversarial accuracy: 33.01 -> 36.23 | Helped |
| Apply CLAT fine-tuning after PGD-AT | DenseNet-121 · ImageNet, PGD-20 | clean accuracy: 63.25 -> 66.10 | Helped |
| Apply CLAT fine-tuning after PGD-AT | DenseNet-121 · CIFAR-100, AutoAttack | adversarial accuracy: 23.13 -> 25.74 | Helped |
| Select critical layers instead of random layers for fine-tuning (ablation) | DenseNet-121 CLAT · CIFAR-10, PGD-10 | adversarial accuracy: 51.35 vs 60.60 | Helped |
| Select critical layers instead of random layers for fine-tuning (ablation) | DenseNet-121 CLAT · CIFAR-100, AutoAttack | adversarial accuracy: 20.21 vs 25.74 | Helped |
| Recompute critical layers dynamically instead of keeping them fixed (ablation) | DenseNet-121 CLAT · CIFAR-10, PGD-10 | adversarial accuracy: 57.01 -> 60.60 | Helped |
| Fine-tune layers with the largest rather than smallest criticality indices (ablation) | DenseNet-121 CLAT · CIFAR-10, PGD-10 | adversarial accuracy: 59.25 vs 60.60 | Helped |
| Use dynamic rather than fixed critical layers (ablation) | ResNet-50 CLAT · CIFAR-100 | adversarial accuracy: 31.89 -> 36.23 | Helped |
Surprising findings
- CLAT applied to clean pretrained models yields non-trivial adversarial robustness (e.g., 39.21% PGD accuracy for DN-121), and running CLAT from scratch eventually surpasses PGD-AT given enough epochs.
- Fine-tuning the least critical layers with the CLAT objective performs worse than fine-tuning the most critical ones but still improves over the vanilla model.
Limitations
- Evaluated only on CNN architectures and image classification; the weakness metric may be imprecise for architectures like Vision Transformers (left to future work).
- Lacks theoretical explanation of why certain layers are critical; the authors state this is open.
- Critical-layer count is chosen by hyperparameter search (about 5% of layers), with per-architecture settings.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Imagenette
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Imagenette AutoAttack RobustBench
Metrics: clean accuracy adversarial accuracy under PGD-20 AutoAttack accuracy black-box accuracy
Baselines: PGD-AT TRADES RiFT LoRA AutoLoRA Fast-AT GradAlign AWP SWA-AT
Builds on: PGD-AT TRADES RiFT Activation Attacks
Closest papers by meaning
- Sparse Fine-Tuning for Parameter-Efficient Adversarial Training (NeurIPS 2026 · similarity 0.90)
- Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks (ICML 2020 · similarity 0.89)
- Attribute-Guided Adversarial Training for Robustness to Natural Perturbations (AAAI 2021 · similarity 0.89)
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.88)
- TAET: Two-Stage Adversarial Equalization Training on Long-Tailed Distributions (CVPR 2025 · similarity 0.88)
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.