Research map / Label-efficient and robust learning / Self-supervised and contrastive
Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes contrastive-equivariant SSL (CE-SSL), which adds an equivariance loss on paired-augmentation difference vectors to standard invariant SSL objectives, and shows it improves ResNet-50 neural predictivity for macaque IT.
Problem
Self-supervised learning objectives produce representations that are overly invariant to input transformations, which may not match structured variability in biological IT representations; the paper asks whether structured transformation-related variability improves alignment with primate IT.
Approach
Converts any invariant SSL loss into a contrastive-equivariant version by applying the same loss to difference vectors between paired, identically augmented views in a second projector space, requiring no access to augmentation parameters and no extra network passes.
Key results
- Barlow Twins with lambda=0.2 CE-SSL ranked 10th of roughly 250 public models on the Brain-Score IT leaderboard at testing time, and equivariant models beat the invariant baseline on all four IT datasets.
- Training the best model for 1000 epochs gave 0.5355 mean fraction of explained variance, reported as the top IT brain prediction model.
- On transfer, CE variants improved ImageNet-100-trained frozen-linear accuracy in 15 of 18 cases but gave limited gains for ImageNet-1k-trained networks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add contrastive-equivariant loss to invariant SSL | ResNet-50 models trained with SimCLR, MMCR, or Barlow Twins · Four macaque IT neural datasets | neural predictivity | Helped |
| Add contrastive-equivariant loss (λ=0.1) | MMCR · ImageNet-100 transfer evaluation across six datasets | top1 accuracy | Mixed |
| Add contrastive-equivariant loss (λ=0.1) | Barlow Twins · ImageNet-100 transfer evaluation across six datasets | top1 accuracy | Mixed |
| Add contrastive-equivariant loss (λ=0.1) | SimCLR · ImageNet-100 transfer evaluation across six datasets | top1 accuracy | Mixed |
| Add contrastive-equivariant loss (λ=0.1) | MMCR, Barlow Twins, or SimCLR · ImageNet-1k transfer evaluation across six datasets | top1 accuracy | Mixed |
| Add contrastive-equivariant loss | ImageNet-100-pretrained models · ImageNet-100 validation | online-linear accuracy | Mixed |
Surprising findings
- Lambda sweep spread in neural predictivity was about 4 times larger than the spread across base invariant objectives, so the equivariance weight mattered more than the SSL objective choice.
- Spatial-Photometric factorization was the only representational metric with correlation above 0.4 across all four IT datasets.
Limitations
- Requires selecting lambda, and each lambda value needs a separate trained network.
- Architecture search over the two projectors was limited by compute; neural predictivity variance over training seeds was not extensively measured.
- Classification performance degrades for lambda above 0.5, and CE-SSL does not improve in-distribution accuracy or most ImageNet-1k transfer results.
Experimental setup
Datasets: ImageNet-1k ImageNet-100 Majaj-Hong Sanghavi-Jozwik Sanghavi Sanghavi-Murty Cifar-10 Cifar-100 Oxford-Pets DTD Flowers-102 Food-101
Benchmarks: Brain-Score IT neural predictivity (4 datasets) frozen-linear transfer on 6 datasets online-linear in-distribution accuracy on ImageNet
Metrics: neural predictivity (ridge regression, noise-ceiled) top-1 accuracy Bures distance coefficient of determination (R2) Pearson correlation
Baselines: SimCLR Barlow Twins MMCR CARE EquiMod SIE Amortised Invariance
Builds on: SimCLR Barlow Twins MMCR CARE EquiMod
Closest papers by meaning
- Understanding the Role of Equivariance in Self-supervised Learning (NeurIPS 2024 · similarity 0.87)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.87)
- In-Context Symmetries: Self-Supervised Learning through Contextual World Models (NeurIPS 2024 · similarity 0.87)
- Interpreting Equivariant Representations (ICML 2024 · similarity 0.87)
- Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning (NeurIPS 2025 · similarity 0.87)
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.