Research map / Label-efficient and robust learning / Self-supervised and contrastive
Understanding the Role of Equivariance in Self-supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives an information-theoretic (explaining-away / synergy) account of why equivariant self-supervised learning such as rotation prediction learns class-relevant features, derives design principles, and shows equivariant backbones (EqResNet18) improve E-SSL on CIFAR.
Problem
Why, when, and how equivariant self-supervised learning (E-SSL), e.g., rotation prediction, learns features that generalize to downstream classification, given that the augmentation is independent of the class label.
Approach
Uses a causal collider structure (class C and augmentation A both parents of augmented input X) to show explaining-away yields I(A;C|X)>0, so predicting the augmentation synergistically encourages encoding class features; derives principles (lossy transformations, class relevance, shortcut pruning), a toy additive model analysis, and shows model equivariance helps.
Key results
- Equivariant ResNet18 raises CIFAR-10 linear probing accuracy from 59.06% to 82.26% under SimCLR augmentation for rotation prediction (ResNet18 baseline 59.06%).
- On CIFAR-100 with SimCLR augmentation, EqResNet18 gains +26.46 points in test classification accuracy over ResNet18 (37.11% vs 10.65%).
- Under aggressive augmentation, replacing crop-only with SimCLR augmentation lifts RotNet test classification accuracy on CIFAR-10 from 57.32% to 59.06%, while training rotation accuracy drops from 97.71% to 83.26%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-10, None augmentation | Test Classification ACC: 56.92 -> 72.32 | Helped |
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-10, Crop&Flip | Test Classification ACC: 57.32 -> 82.54 | Helped |
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-10, SimCLR | Test Classification ACC: 59.06 -> 82.26 | Helped |
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-100, None augmentation | Test Classification ACC: 11.31 -> 32.38 | Helped |
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-100, Crop&Flip | Test Classification ACC: 13.19 -> 49.47 | Helped |
| use a p4-equivariant backbone instead of ResNet18 | rotation prediction · CIFAR-100, SimCLR | Test Classification ACC: 10.65 -> 37.11 | Helped |
| use SimCLR augmentations instead of no augmentation or Crop&Flip | rotation prediction with ResNet18 · CIFAR-10 | Test Cls ACC: 56.92 -> 59.06; 57.32 -> 59.06 | Helped |
| use CARE loss instead of CE loss | rotation prediction · CIFAR-10 | Test Cls ACC: 57.32 -> 64.50 | Helped |
Surprising findings
- Equivariance alone is not sufficient: several transformations (e.g., horizontal flip, grayscale) yield near-random linear probing despite low or near-zero equivariance loss.
- Equivariant backbone models gave gains of over 20 points on CIFAR-10 even with mild augmentation where rotation prediction is already nearly perfect.
Limitations
- Theory relies on Shannon information with a simplified additive data model for quantitative results; V-information extension is only qualitative.
- Experiments are limited to small CIFAR-scale datasets and ResNet backbones with linear probing.
- Authors state they did not extensively explore better E-SSL designs.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet-200
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet-200
Metrics: linear probing test accuracy training rotation prediction accuracy
Baselines: RotNet SimCLR CARE Random initialized encoder
Builds on: RotNet SimCLR V-information CARE
Closest papers by meaning
- Equivariant Self-Supervised Learning: Encouraging Equivariance in Representations (ICLR 2022 · similarity 0.93)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.87)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.87)
- Self-supervised Transformation Learning for Equivariant Representations (NeurIPS 2024 · similarity 0.86)
- Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning (ICLR 2024 · similarity 0.85)
- Interpreting Equivariant Representations (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.