Research map / Adversarial robustness and security / Certified and empirical robustness
3SAT: A Simple Self-Supervised Adversarial Training Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3SAT, a self-supervised adversarial training framework that adversarially trains on unaugmented samples with a dynamic warm-up-based scheduling of the AT loss, reaching state-of-the-art robustness on CIFAR-10 and CIFAR-100.
Problem
Self-supervised adversarial training (self-AT) lags behind supervised adversarial training in robustness, and data augmentation used for positive pairs hurts robustness while joint SSL+AT optimization is hard and can collapse.
Approach
Perform adversarial training on original unaugmented samples versus their adversarial counterparts (one adversarial example per step, no dual BN), and gradually ramp the AT loss weight after an SSL warm-up (dynamic training objective scheduling) to avoid collapse.
Key results
- On CIFAR-10 under AFF evaluation, 3SAT reaches 66.79% AutoAttack accuracy vs 50.60% for prior best self-AT (DynACL-AIR) and 55.54% for supervised AT.
- 3SAT achieves 93.55% standard accuracy under AFF on CIFAR-10, versus 82.14% for DynACL-AIR.
- 3SAT halves pre-training time (14.28 h vs 29.4 h for DynACL) on a single RTX3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adding warm-up phase (200 epochs) for the self-supervised part before adversarial training (dynamic training objective scheduling) (ablation) | 3SAT on CIFAR-100 · CIFAR-100 | AA(%) and SA(%): SA +6.83 (51.06→57.89); AA +3.08 (24.96→28.04) | Helped |
| Increasing warm-up epochs beyond 200 (to 400-800) (ablation) | 3SAT on CIFAR-100 · CIFAR-100 | AA(%): AA 28.04 (200) vs 29.01 (400), 28.86 (600), 27.14 (800); SA rises to 60.68 at 800 | No clear effect |
| Using original (unaugmented) samples for adversarial training instead of augmented samples | 3SAT self-AT · CIFAR-10 | AA(%) under SLF (robustness): 3SAT 49.73 vs DynACL-AIR 45.17 (+4.56) | Helped |
| Adversarial training on original samples with dynamic scheduling, compared to prior self-AT (DynACL-AIR) | self-AT on CIFAR-10 under AFF · CIFAR-10 | AA(%): 66.79 vs 50.60 (+16.19); beats sup-AT 55.54 by 11.25 | Helped |
| Adversarial training on augmented samples (narrowing distance between augmented and adversarial samples) instead of original samples (ablation) | self-AT training · CIFAR-10 training loss/stability | adversarial training loss and training stability | Hurt |
| Removing dual BN paths (single BN) in 3SAT encoder | 3SAT · CIFAR-10 | SA(%) and AA(%) | No clear effect |
| Generating one adversarial sample per AT iteration instead of two (3SAT vs DynACL) | self-AT pretraining · CIFAR-10 pretraining duration | training duration (h): 14.28 h vs 29.4 h for DynACL | Helped |
| Semi-supervised fine-tuning with 1% labels, 3SAT vs DynACL++ (AFF) | 3SAT · CIFAR-10 1% labels | AFF AA(%): 52.25 vs 46.95 (DynACL++ at 1%) | Helped |
Surprising findings
- Self-AT with 3SAT outperforms supervised AT in robust accuracy on CIFAR-10 with the same ResNet-18 architecture.
- Adversarially training on augmented positive pairs causes high AT loss and collapse of self-supervised training, while original samples do not.
Limitations
- Warm-up length must be tuned per dataset (W=0 for CIFAR-10/STL-10, 200 for CIFAR-100).
- Only 5 PGD steps used during pre-training to speed convergence.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 STL-10
Benchmarks: CIFAR-10 CIFAR-100 STL-10
Metrics: AutoAttack robust accuracy standard accuracy
Baselines: RoCL ACL AdvCL TAROSS DynACL DynACL-AIR DynACL++ sup-AT (Sehwag et al. 2021)
Builds on: BYOL ACL DynACL Madry adversarial training solo-learn
Closest papers by meaning
- Self-Supervised Adversarial Training via Diverse Augmented Queries and Self-Supervised Double Perturbation (NeurIPS 2024 · similarity 0.92)
- Annealing Self-Distillation Rectification Improves Adversarial Training (ICLR 2024 · similarity 0.88)
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.87)
- Long-tailed Adversarial Training with Self-Distillation (ICLR 2025 · similarity 0.87)
- Explaining the role of Intrinsic Dimensionality in Adversarial Training (ICML 2025 · similarity 0.86)
- Toward Improving Robustness and Accuracy in Unsupervised Domain Adaptation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.