Research map / Label-efficient and robust learning / Self-supervised and contrastive
Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adv-SSL, a min-max self-supervised method that removes the mini-batch bias of covariance-identity regularization via an adversarial matrix variable G, and proves end-to-end few-shot error bounds for it.
Problem
Covariance/correlation-identity regularized self-supervised learning uses a biased sample estimate of the population regularizer (Frobenius norm of expectation minus identity), causing optimization drift in mini-batches and blocking end-to-end theory for downstream few-shot classification.
Approach
Rewrites the regularizer as a supremum over an auxiliary matrix G in a Frobenius ball, giving an unbiased sample-level estimator; training becomes an alternating min-max where G is solved in closed form and the encoder is updated with G detached, and the paper derives a misclassification bound that depends on unlabeled source size and downstream class counts.
Key results
- Adv-SSL reaches 93.01% linear / 90.97% k-NN on CIFAR-10, versus 92.47% / 90.19% for LogDet, the strongest listed baseline.
- On CIFAR-100 Adv-SSL gets 68.94% linear and 58.50% k-NN, above LogDet's 67.32% / 57.56%.
- Adv-SSL trains slightly faster and with similar memory than Barlow Twins (e.g., 51s vs 68s per epoch on CIFAR-10), since the inner maximization has a closed form.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Adv-SSL instead of LogDet | self-supervised representation learning · CIFAR-10 linear | accuracy: 92.47 -> 93.01 | Helped |
| use Adv-SSL instead of LogDet | self-supervised representation learning · CIFAR-10 k-nn | accuracy: 90.19 -> 90.97 | Helped |
| use Adv-SSL instead of WMSE4 | self-supervised representation learning · CIFAR-100 linear | accuracy: 67.64 -> 68.94 | Helped |
| use Adv-SSL instead of LogDet | self-supervised representation learning · CIFAR-100 k-nn | accuracy: 57.56 -> 58.50 | Helped |
| use Adv-SSL instead of BYOL | self-supervised representation learning · Tiny ImageNet linear | accuracy: 51.00 -> 50.21 | Hurt |
| use Adv-SSL instead of BYOL | self-supervised representation learning · Tiny ImageNet k-nn | accuracy: 36.24 -> 37.40 | Helped |
| add the explicit alignment term (ablation) | Adv-SSL · CIFAR-10 linear | accuracy: 92.42 -> 93.01 | Helped |
| add the explicit alignment term (ablation) | Adv-SSL · CIFAR-10 k-nn | accuracy: 90.01 -> 90.97 | Helped |
| add the explicit alignment term (ablation) | Adv-SSL · CIFAR-100 linear | accuracy: 67.27 -> 68.94 | Helped |
| add the explicit alignment term (ablation) | Adv-SSL · CIFAR-100 k-nn | accuracy: 58.10 -> 58.50 | Helped |
| remove horizontal flipping from the augmentation set (ablation) | Adv-SSL · CIFAR-10 | accuracy: 93.01 -> 91.03 (Linear); 90.97 -> 88.34 (k-nn) | Hurt |
| remove color distortion and horizontal flipping (ablation) | Adv-SSL · CIFAR-10 | accuracy: 93.01 -> 89.18 (Linear); 90.97 -> 85.65 (k-nn) | Hurt |
Surprising findings
- Adv-SSL without the explicit alignment term still reaches 92.42% linear on CIFAR-10, close to the full method.
- Tiny ImageNet linear accuracy of Adv-SSL (50.21%) is below BYOL (51.00%).
Limitations
- Theory relies on strong assumptions (Hölder smoothness of encoder, Lipschitz augmentations, domain shift rates, and augmentation quality conditions) that may not hold in practice.
- Experiments are limited to small image benchmarks (CIFAR and Tiny ImageNet) with a ResNet-18 backbone and a single V100 GPU.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet
Metrics: top-1 accuracy (linear probe) top-1 accuracy (k-NN, k=5) training memory training time per epoch
Baselines: Barlow Twins Beyond Separability SimCLR BYOL WMSE VICReg LogDet
Builds on: Barlow Twins Beyond Separability SimCLR
Closest papers by meaning
- Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.86)
- Self-Supervised Adversarial Training via Diverse Augmented Queries and Self-Supervised Double Perturbation (NeurIPS 2024 · similarity 0.84)
- 3SAT: A Simple Self-Supervised Adversarial Training Framework (AAAI 2025 · similarity 0.84)
- Self-Supervised Dataset Distillation for Transfer Learning (ICLR 2024 · similarity 0.84)
- Revisiting Adversarial Training at Scale (CVPR 2024 · similarity 0.83)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.