Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervised Transformation Learning for Equivariant Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-supervised Transformation Learning (STL), which replaces transformation labels in equivariant learning with image-invariant transformation representations learned from image pairs under aligned transformed batches.
Problem
Explicit equivariant representation learning relies on transformation labels, which treat transformations independently, ignore interdependencies, and cannot express complex transformations such as AugMix.
Approach
Derive transformation representations from pairs of representations (original and transformed), train them with a contrastive loss to be invariant to the image (aligning same-transformation pairs from different images), and use them as conditioning for an equivariant transformation network in place of labels.
Key results
- On ImageNet100-pretrained ResNet-50, STL outperforms prior methods on 7 of 11 out-domain linear classification datasets, and with AugMix reaches a mean of 66.11% across the 11 datasets.
- On VOC07+12 detection, STL with AugMix reaches 52.70 AP_all versus 51.55 for EquiMod and 45.67 for SimCLR.
- Transformation type classification accuracy on STL10 ResNet-18 rises to 93.67% for STL versus 88.49% for AugSelf and 68.54% for SimCLR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STL instead of SimCLR | ImageNet100-pretrained ResNet-50 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 60.35 -> 64.18 | Helped |
| Add AugMix to STL | ImageNet100-pretrained ResNet-50 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 64.18 -> 66.11 | Helped |
| Use STL instead of SimCLR | ImageNet100-pretrained ResNet-50 · In-domain classification | linear evaluation accuracy (%): 81.20 -> 81.10 | Hurt |
| Add AugMix to STL | ImageNet100-pretrained ResNet-50 · In-domain classification | linear evaluation accuracy (%): 81.10 -> 81.64 | Helped |
| Use STL instead of EquiMod | ImageNet100-pretrained ResNet-50 · VOC07+12 object detection | AP_all: 51.55 -> 51.95 | Helped |
| Add AugMix to STL | ImageNet100-pretrained ResNet-50 · VOC07+12 object detection | AP_all: 51.95 -> 52.70 | Helped |
| Add self-supervised transformation learning loss (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 48.49 -> 49.97 | Helped |
| Add invariant learning loss (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 48.62 -> 49.97 | Helped |
| Add equivariant learning loss (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 47.30 -> 49.97 | Helped |
| Use crop + color instead of crop as equivariance targets (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 47.70 -> 49.97 | Helped |
| Use crop + color instead of color as equivariance targets (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 48.51 -> 49.97 | Helped |
| Use all transformations instead of crop + color as equivariance targets (ablation) | STL10-pretrained ResNet-18 · Out-domain classification (11-dataset mean) | linear evaluation accuracy (%): 49.97 -> 48.48 | Hurt |
Surprising findings
- Transformation prediction accuracy for AugSelf (88.49%) is outperformed by STL despite AugSelf explicitly regressing transformation parameters.
- Adding AugMix to SimCLR alone lowers in-domain STL10 accuracy on ImageNet100 (81.20 to 80.54), while STL with AugMix improves in-domain accuracy.
Limitations
- Cannot handle transformations involving multiple images, such as mixup, since it relies on pairwise transformation representations.
- Gains on some out-domain tasks are modest; STL is not best on all 11 classification datasets (7 of 11).
Experimental setup
Datasets: STL10 ImageNet100 CIFAR10 CIFAR100 Food MIT67 Pets Flowers Caltech101 Cars Aircraft DTD
Benchmarks: CIFAR10 CIFAR100 Food MIT67 Pets Flowers Caltech101 Cars Aircraft DTD SUN397 STL10
Metrics: top-1 accuracy mean per-class accuracy average precision MSE MRR Hit@k precision
Baselines: SimCLR AugSelf E-SSL SEN EquiMod SIE BYOL SimSiam Barlow Twins
Builds on: SimCLR EquiMod AugSelf SIE AugMix
Closest papers by meaning
- Self-Supervised Learning Based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation (AAAI 2026 · similarity 0.92)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.86)
- Modulate Your Spectrum in Self-Supervised Learning (ICLR 2024 · similarity 0.86)
- Understanding the Role of Equivariance in Self-supervised Learning (NeurIPS 2024 · similarity 0.86)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.85)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.