Research map / Label-efficient and robust learning / Self-supervised and contrastive
In-Context Symmetries: Self-Supervised Learning through Contextual World Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ContextSSL, a transformer-based contextual world model that learns adaptive invariance or equivariance to image transformations depending on a few in-context examples, without parameter updates at adaptation time.
Problem
Self-supervised vision methods hard-code invariances or equivariances to fixed augmentations, producing brittle representations that do not transfer to downstream tasks whose required symmetries differ.
Approach
Treat transformations as actions in a world model and feed a context of (input, action, transformed output) triplets to a decoder-only transformer; including action parameters in the context yields equivariance, excluding them yields invariance. A context mask prevents shortcut learning and an auxiliary transformation predictor prevents collapse to invariance.
Key results
- On 3DIEBench with rotation context, ContextSSL reaches rotation prediction R^2 of 0.744 at context 126 versus 0.671 for SIE and 0.633 for SEN, while retaining 80.4% classification top-1.
- With color context, ContextSSL reaches color prediction R^2 of 0.986 versus 0.975 for SIE and 0.949 for SEN.
- On rotation nearest-neighbour retrieval, ContextSSL H@1 rises from 0.108 at context 0 to 0.292 at context 126, versus 0.05 for EquiMOD and SEN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase rotation-context length from 0 to 126 | ContextSSL · 3DIEBench rotation prediction | R²: 0.734 -> 0.744 | Helped |
| Increase rotation-context length from 0 to 126 | ContextSSL · 3DIEBench color prediction | R²: 0.908 -> 0.046 | Hurt |
| Increase color-context length from 0 to 126 | ContextSSL · 3DIEBench color prediction | R²: 0.908 -> 0.986 | Helped |
| Increase color-context length from 0 to 126 | ContextSSL · 3DIEBench rotation prediction | R²: 0.735 -> 0.344 | Hurt |
| Increase rotation-context length from 0 to 126 | ContextSSL · 3DIEBench validation | MRR: 0.240 -> 0.402 | Helped |
| Increase rotation-context length from 0 to 126 | ContextSSL · 3DIEBench validation | H@1: 0.108 -> 0.292 | Helped |
| Use rotation context rather than none context | ContextSSL · 3DIEBench rotation prediction | R²: 0.473 -> 0.738 | Helped |
| Remove the context mask (ablation) | SimCLR+ · 3DIEBench classification | top-1: 81.0 -> 42.3 | Hurt |
| Remove the context mask (ablation) | SimCLR+ · CIFAR-10 classification | top-1: 89.7 -> 59.9 | Hurt |
| Add the auxiliary predictor (ablation) | ContextSSL · 3DIEBench rotation prediction | R²: 0.227 -> 0.744 | Helped |
| Add the auxiliary predictor (ablation) | ContextSSL · 3DIEBench color prediction | R²: -0.004 -> 0.046 | Helped |
| Set random masking probability to 0.90 rather than 0.00 (ablation) | ContextSSL · 3DIEBench rotation context, rotation prediction | R²: 0.677 $\pm$ 0.004 -> 0.734 $\pm$ 0.002 | Helped |
Surprising findings
- ContextSSL is already equivariant to rotation and color at zero context, outperforming equivariant baselines without any context.
- Without the context mask, SimCLR+ classification accuracy drops from 81.0 to 42.3 on 3DIEBench.
Limitations
- Symmetries are limited to hand-crafted transformations; naturally occurring symmetries are not explored.
- Invariance performance degrades at large context lengths, attributed to the 90% random masking ratio causing out-of-distribution generalization.
- Classification accuracy of ContextSSL is slightly below SimCLR (80.4 vs 85.3 on 3DIEBench; 88.5 vs 89.1 on CIFAR-10).
Experimental setup
Datasets: 3DIEBench CIFAR-10
Benchmarks: 3DIEBench CIFAR-10
Metrics: top-1 classification accuracy R^2 MRR H@1 H@5
Baselines: SimCLR VICReg EquiMOD SEN SIE
Builds on: SimCLR VICReg Image World Models EquiMOD SIE
Closest papers by meaning
- Understanding the Role of Equivariance in Self-supervised Learning (NeurIPS 2024 · similarity 0.87)
- seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models (NeurIPS 2025 · similarity 0.87)
- From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication (ICLR 2024 · similarity 0.87)
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.87)
- Neural Isometries: Taming Transformations for Equivariant ML (NeurIPS 2024 · similarity 0.87)
- Self-supervised Transformation Learning for Equivariant Representations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.