Research map / Label-efficient and robust learning / Self-supervised and contrastive
State Representation Learning Using an Unbalanced Atlas
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIM-UA, an unbalanced-atlas (multi-head manifold) self-supervised objective that adapts ST-DIM and SimCLR, improving linear probe F1 on AtariARI at large hidden-unit counts.
Problem
Manifold-based self-supervised representations (MSimCLR) only beat baselines at extremely low encoding dimensions, and uniform-prior multi-head atlases suffer high prediction uncertainty and head collapse, limiting use with large embeddings.
Approach
Drop the uniform prior over chart memberships, train membership probabilities to be far from uniform via an MMD-style loss, and use averaged (dilated) prediction targets across heads in the global-local score function so that head embeddings do not converge, enabling scaling to many hidden units.
Key results
- DIM-UA reaches mean probe F1 0.75 on AtariARI at 16384 total units versus 0.72 for ST-DIM at 256 units and 0.70 for ST-DIM* at 16384 units.
- On CIFAR10, SimCLR-UA with 8 heads of 512 units reaches 88.6% linear accuracy, versus 88.3% for SimCLR and 86.4% best MSimCLR at 8 heads.
- ST-DIM* with a single 16384-unit head collapses on Freeway (F1 0.3 with std 0.355), while DIM-UA achieves 0.86.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the unbalanced atlas paradigm with four heads of 4096 units each | ST-DIM · AtariARI (19-game mean) | F1: 0.7 ± 0.033 -> 0.75 ± 0.013 | Helped |
| Increase a single ST-DIM output head to 16384 units (ablation) | ST-DIM · AtariARI (19-game mean) | F1: 0.72 -> 0.7 ± 0.033 | Hurt |
| Apply the unbalanced atlas paradigm | SimCLR · CIFAR10 | accuracy: 0.883 ± 0.002 -> 0.886 ± 0.002 | Helped |
| Use clamping and FC2 together (ablation) | SimCLR-UA (4 heads, 512 units each) · CIFAR10 | accuracy: 0.883 -> 0.884 | Helped |
| Add FC1 to the clamping + FC2 configuration (ablation) | SimCLR-UA (4 heads, 512 units each) · CIFAR10 | accuracy: 0.884 -> 0.872 | Hurt |
| Use linear scaling with τ=0.2 (ablation) | SimCLR-UA · CIFAR10 (ResNet18) | accuracy: 0.791 -> 0.797 | Helped |
| Use linear scaling with τ=0.1 (ablation) | SimCLR-UA · CIFAR10 (ResNet18) | accuracy: 0.791 -> 0.8 | Helped |
| Use linear scaling with τ=0.05 (ablation) | SimCLR-UA · CIFAR10 (ResNet18) | accuracy: 0.799 -> 0.796 | Hurt |
| Use linear scaling with τ=0.02 (ablation) | SimCLR-UA · CIFAR10 (ResNet18) | accuracy: 0.802 -> 0.785 | Hurt |
| Use eight heads rather than two heads (ablation) | DIM-UA · AtariARI, total units below 2048 | F1 | Hurt |
| Use eight heads rather than two heads (ablation) | DIM-UA · AtariARI, total units 16384 | F1 | Helped |
| Remove dilated prediction targets (ablation) | DIM-UA (-UA) · AtariARI, 512 units | F1 | Helped |
Surprising findings
- Scaling a single head to 16384 units does not reliably help (ST-DIM* mean F1 0.70 vs ST-DIM 0.72), with 7 of 19 games worse.
- Improvement of SimCLR-UA over SimCLR on CIFAR10 is small (about 0.3 points at best).
Limitations
- DIM-UA underperforms ST-DIM at small hidden-unit counts because multiple heads converge more slowly.
- Convexity assumption required for the Minkowski-sum argument is not guaranteed in practice.
- CIFAR10 gains over SimCLR are small and rely on modified hyperparameters.
Experimental setup
Datasets: AtariARI CIFAR10
Benchmarks: AtariARI CIFAR10
Metrics: probe F1 probe accuracy linear evaluation accuracy
Baselines: VAE CPC ST-DIM MSimCLR SimCLR
Builds on: ST-DIM MSimCLR Deep InfoMax SimCLR InfoNCE
Closest papers by meaning
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.83)
- Robust Visual Recognition with Class-Imbalanced Open-World Noisy Data (AAAI 2024 · similarity 0.83)
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold (NeurIPS 2025 · similarity 0.83)
- Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement (NeurIPS 2024 · similarity 0.83)
- Improve Representation for Imbalanced Regression through Geometric Constraints (CVPR 2025 · similarity 0.83)
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.