Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervised Representation Learning from Random Data Projectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning from randomness (LFR), a modality- and architecture-agnostic SSRL method that trains an encoder to reconstruct outputs of diverse random data projectors via batch-wise Barlow Twins loss, without augmentations or masking.
Problem
Self-supervised representation learning typically relies on data augmentations or masking that are hard to design for non-vision/NLP modalities and can conflict with application-specific constraints (e.g., medical images, time series, tabular data).
Approach
Learn representations by predicting outputs of randomly initialized, scaled-down network projectors from the encoder's embedding through simple predictors; diverse projectors are chosen from many candidates via a determinantal point process, and an EM-style alternating optimization with a batch-wise Barlow Twins divergence is used.
Key results
- LFR achieves 93.1% on HAR and 97.9% on Epilepsy, outperforming TS-TCC (91.2%, 97.6%) and other SSRL baselines in the time series benchmarks.
- On Kvasir medical images LFR reaches 74.9% accuracy, the best among self-supervised methods (DACL 72.0%, SimCLR 72.1%).
- On CIFAR10 LFR gets 64.3%, behind SimSiam (89.2%) and SimCLR (86.7%), which rely on heavy natural-image augmentations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn representations by predicting random data projections | LFR · HAR | accuracy: 91.2{ $\pm$0.8} -> 93.1{ $\pm$0.5} | Helped |
| learn representations by predicting random data projections | LFR · Epilepsy | accuracy: 97.6{ $\pm$0.2} -> 97.9{ $\pm$0.2} | Helped |
| learn representations by predicting random data projections | LFR · MIMIC-III | Cohen's Kappa: 44.9{ $\pm$0.5} -> 46.6{ $\pm$0.3} | Helped |
| learn representations by predicting random data projections | LFR · Income | accuracy: 85.0{ $\pm$0.1} -> 85.2{ $\pm$0.1} | Helped |
| learn representations by predicting random data projections | LFR · Theorem | accuracy: 50.7{ $\pm$0.7} -> 51.6{ $\pm$0.7} | Helped |
| learn representations by predicting random data projections | LFR · HEPMASS | accuracy: \textbf{90.7{ $\pm$0.0}} -> 90.1{ $\pm$0.2} | Hurt |
| learn representations by predicting random data projections | LFR · Kvasir | accuracy: 72.4{ $\pm$0.6} -> 74.9{ $\pm$0.6} | Helped |
| replace Batch-wise Barlow Twins loss with InfoNCE loss (ablation) | LFR · Kvasir | accuracy: 74.9 \pm 0.6 -> 72.6 \pm 0.4 | Hurt |
| encourage diversity in projectors during initialization and selection (ablation) | LFR · Kvasir | accuracy | Helped |
| increase the number of random projectors up to 6 (ablation) | LFR · Kvasir | linear probing accuracy | Helped |
| increase the number of random projectors beyond 6 (ablation) | LFR · Kvasir | linear probing accuracy | Hurt |
| vary the training batch size from 64 to 512 (ablation) | LFR · Kvasir | linear accuracy | No clear effect |
Surprising findings
- Predicting purely random data projections yields representations competitive with or better than augmentation-based methods on several non-vision domains, including outperforming supervised training on Income.
- The authors found a bug in the official TS-TCC implementation (fixed augmentation once), and reported corrected baseline numbers.
Limitations
- Performs worse than augmentation-based methods on natural images (CIFAR10), where heavy augmentations are well designed.
- Not best on HEPMASS, where it is comparable to autoencoder and SCARF rather than superior.
- Hyperparameters such as number of projectors and diversity selection add design choices; scope limited to classification tasks with frozen-encoder linear/MLP probes.
Experimental setup
Datasets: HAR Epilepsy MIMIC-III Income Theorem HEPMASS Kvasir CIFAR10 Fault Diagnosis
Benchmarks: HAR Epilepsy MIMIC-III Income Theorem HEPMASS Kvasir CIFAR10 Fault Diagnosis
Metrics: accuracy Cohen's Kappa linear probe accuracy
Baselines: Autoencoder DACL DIET SimSiam SimCLR TS-TCC SCARF STab Supervised Random Init
Builds on: Barlow Twins SimCLR SimSiam DIET TS-TCC
Closest papers by meaning
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.87)
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections (NeurIPS 2025 · similarity 0.87)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.86)
- Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.86)
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.86)
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.