Research map / Label-efficient and robust learning / Self-supervised and contrastive
Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARE, a contrastive method that adds an orthogonal-equivariance loss so that input augmentations act as rotations on the sphere embedding, combined with InfoNCE for approximate invariance.
Problem
Standard contrastive self-supervised learning enforces invariance to augmentations, discarding features (e.g., color) that matter for downstream tasks, while existing equivariant approaches use non-linear, non-composable transformations of embeddings.
Approach
Train an encoder so that the same augmentation applied to different inputs produces the same change in embedding angles, which theoretically forces augmentations to act as orthogonal (rotation/reflection) maps on the sphere; combine this equivariance loss with InfoNCE's invariance and uniformity terms, weighted by lambda.
Key results
- CARE with SimCLR and MoCo-v2 yields consistent linear-probe accuracy gains over SimCLR on CIFAR10, CIFAR100, STL10 and ImageNet100 (averaged over 5 seeds).
- CARE achieves lower Wahba's problem error and much lower relative rotational equivariance error than SimCLR on CIFAR10 with ResNet-18.
- Equivariance loss alone collapses, and uniformity plus equivariance without invariance yields non-trivial but weaker features than InfoNCE-based baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add orthogonal equivariance loss to SimCLR | ResNet-50 · CIFAR10 linear probe | top-1 accuracy | Helped |
| Add orthogonal equivariance loss to SimCLR | ResNet-50 · CIFAR100 linear probe | top-1 accuracy | Helped |
| Add orthogonal equivariance loss to SimCLR | ResNet-50 · STL10 linear probe | top-1 accuracy | Helped |
| Add orthogonal equivariance loss to SimCLR | ResNet-50 · ImageNet100 linear probe | top-1 accuracy | Helped |
| Add orthogonal equivariance loss to MoCo-v2 | ResNet-50 · ImageNet100 linear probe | top-1 accuracy | Helped |
| Add orthogonal equivariance loss to SimCLR | ResNet-18 · CIFAR10 Wahba's problem | Wahba error | Helped |
| Add orthogonal equivariance loss to SimCLR | ResNet-18 · CIFAR10 relative rotational equivariance | relative equivariance error | Helped |
| Add orthogonal equivariance loss to SimCLR | image retrieval model · Flowers-102 color-jitter retrieval | sensitivity to color changes | Helped |
| Remove the uniformity non-collapse term while retaining invariance and equivariance losses (ablation) | CARE · CIFAR10 | representation collapse | Hurt |
| Use only the uniformity loss (ablation) | CARE · CIFAR10 | linear-probe performance | Hurt |
| Add orthogonal equivariance loss to the uniformity loss (ablation) | uniformity-only model · CIFAR10 | representation non-collapse | Helped |
| Add invariance loss to uniformity plus orthogonal equivariance losses (ablation) | uniformity-plus-equivariance model · CIFAR10 | linear-probe performance | Helped |
Surprising findings
- Equivariance loss combined only with uniformity gives non-trivial representations but underperforms invariance-based contrastive baselines, while adding the invariance term yields the best results.
- Optimal performance requires approximate rather than exact invariance; CARE is intentionally less invariant (higher invariance loss) than SimCLR.
Limitations
- Recovering the rotation for a given augmentation requires solving Wahba's problem per augmentation rather than being directly identifiable.
- Only approximate equivariance is achieved, not exact equivariance.
- Experiments are limited to image datasets and ResNet backbones at moderate scale.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 ImageNet100 Flowers-102
Benchmarks: CIFAR10 CIFAR100 STL10 ImageNet100 Flowers-102
Metrics: top-1 linear probe accuracy Wahba's problem error relative rotational equivariance gamma cosine similarity variance top-5 retrieval sensitivity
Baselines: SimCLR MoCo-v2 InfoNCE-based invariance uniformity + equivariance only random initialization
Builds on: SimCLR MoCo-v2 InfoNCE Wahba's problem / Kabsch algorithm Lie-group / learned equivariance (Dehmamy et al. style approaches)
Closest papers by meaning
- Equivariance versus Augmentation for Spherical Images (ICML 2022 · similarity 0.87)
- Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions (NeurIPS 2025 · similarity 0.87)
- Neural Isometries: Taming Transformations for Equivariant ML (NeurIPS 2024 · similarity 0.86)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.86)
- From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication (ICLR 2024 · similarity 0.85)
- Grounding Continuous Representations in Geometry: Equivariant Neural Fields (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.