Research map / Label-efficient and robust learning / Self-supervised and contrastive
URLOST: Unsupervised Representation Learning without Stationarity or Topology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes URLOST, which clusters signal dimensions via density-adjusted spectral clustering on mutual information, aligns clusters with a self-organizing layer, and trains a masked autoencoder, enabling representation learning on data without known stationarity or topology.
Problem
Self-supervised representation learning typically assumes data has known stationarity and domain topology (e.g., image grids), which fails for irregular, non-stationary high-dimensional signals such as foveated vision, V1 neural recordings, and gene expression.
Approach
Recover a coarse topology from pairwise mutual information between signal dimensions via density-adjusted spectral clustering, then align the resulting irregular clusters with a learnable non-shared self-organizing layer before a standard masked autoencoder with a ViT backbone.
Key results
- URLOST MAE reaches 86.4% on Permuted CIFAR-10 and 85.4% on Foveated CIFAR-10 linear probing, while pixel-level MAE gets 56.7% and 48.5% and SimCLR (ResNet-18) gets 47.9% and 38.0%.
- On V1 response decoding, URLOST MAE achieves 78.75% versus 75.64% for beta-VAE, 73.9% for raw signal, and 70.6% for MAE.
- On TCGA pan-cancer classification, URLOST MAE achieves 94.90% versus 94.15% for beta-VAE, 91.7% for raw signal, and 90.6% for MAE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use similar-dimension clusters as masking units instead of randomly aggregating dimensions (ablation) | URLOST MAE · Permuted CIFAR-10 | probing accuracy: 55.7% -> 86.4% | Helped |
| use similar-dimension clusters as masking units instead of individual dimensions (ablation) | URLOST MAE · Foveated CIFAR-10 | probing accuracy: 48.5% -> 85.4% | Helped |
| use similar-dimension clusters as masking units instead of randomly aggregating dimensions (ablation) | URLOST MAE · TCGA Gene | probing accuracy: 91.7% -> 94.9% | Helped |
| use similar-dimension clusters as masking units instead of individual dimensions (ablation) | URLOST MAE · V1 Response | probing accuracy: 64.8% -> 78.8% | Helped |
| use a non-shared self-organizing projection instead of a shared projection (ablation) | URLOST MAE · Locally-Permuted CIFAR-10 | linear probing accuracy: 81.4 % -> 87.6 % | Helped |
| use a non-shared self-organizing projection instead of a shared projection (ablation) | URLOST MAE · Permuted CIFAR-10 | linear probing accuracy: 80.7 % -> 86.4 % | Helped |
| use density-adjusted spectral clustering instead of uniform-density spectral clustering (ablation) | URLOST MAE · Foveated CIFAR-10 | linear probing accuracy: 82.7 % -> 85.4 % | Helped |
| set α=0.5 and β=2 instead of α=1.0 and β=2 (ablation) | URLOST MAE · Foveated CIFAR-10 | evaluation accuracy: 81.62 % -> 85.43 % | Helped |
| set α=0.5 and β=2 instead of α=0 and β=0 (ablation) | URLOST MAE · Foveated CIFAR-10 | evaluation accuracy: 82.74 % -> 85.43 % | Helped |
Surprising findings
- On standard CIFAR-10, SimCLR with ResNet-18 (90.7%) and MAE with ViT patches (88.3%) outperform URLOST-style clustering inputs, so the gains are confined to non-stationary/non-topological settings.
- Clustering on the mutual information graph of 10,000-dimensional data takes about 130 minutes, indicating quadratic scaling limits for very high-dimensional inputs.
Limitations
- Clustering and mutual information are computed separately from MAE training and scale quadratically, infeasible for d > 10,000 with current implementation.
- Cluster generation is not learned end-to-end with the representation model.
- Evaluated only on relatively small natural science datasets; hyperparameters (cluster count, density parameters) need per-dataset grid search.
Experimental setup
Datasets: CIFAR-10 Foveated CIFAR-10 Permuted CIFAR-10 Locally-Permuted CIFAR-10 V1 neural response dataset TCGA miRNA pan-cancer dataset
Benchmarks: CIFAR-10 Permuted CIFAR-10 Foveated CIFAR-10 V1 response decoding TCGA pan-cancer classification
Metrics: linear probing accuracy decoding accuracy classification accuracy (5-fold CV)
Baselines: MAE SimCLR beta-VAE Vision GNN (ViG) ResNet-18
Builds on: masked autoencoder SimCLR beta-VAE Vision Transformer FlexiViT
Closest papers by meaning
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.87)
- Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning (CVPR 2022 · similarity 0.87)
- Self-Organizing Visual Prototypes for Non-Parametric Representation Learning (ICML 2025 · similarity 0.85)
- Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels (ICLR 2024 · similarity 0.85)
- Let Go of Your Labels with Unsupervised Transfer (ICML 2024 · similarity 0.85)
- Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.