Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-Supervised Representation Learning with Meta Comprehensive Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds a plug-and-play CompMod module with Meta Comprehensive Regularization (bi-level optimization plus maximum-entropy coding) to existing SSL methods so that learned representations capture more task-relevant, non-shared information.
Problem
Data augmentation in self-supervised learning enforces invariance across views, so only the shared information is captured and label-related information lost to augmentation is discarded, hurting downstream generalization.
Approach
Fuse the embeddings of two augmented views into a comprehensive embedding, regularize each view toward it via a maximum-entropy coding (lossy coding rate, Taylor-expanded) loss, and optimize the comprehensive projection head with a bi-level scheme; the approach is added on top of SimCLR, BYOL and Barlow Twins.
Key results
- BYOL+ reaches 93.96% linear accuracy on CIFAR-10 (ResNet-18), versus 91.73% for BYOL.
- Barlow Twins+ achieves about 4% top-1 gain on ImageNet-100 over BarlowTwins (76.88 vs 72.88).
- BYOL+ achieves 39.7 AP object detection and 35.4 AP instance segmentation on COCO, above BYOL (37.9 / 33.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CompMod and MCR to BYOL | BYOL, ResNet-18 · CIFAR-10 | linear accuracy: 91.73 -> 93.96 | Helped |
| Add CompMod and MCR to SimCLR | SimCLR, ResNet-18 · CIFAR-100 | linear accuracy: 66.83 -> 68.91 | Helped |
| Add CompMod and MCR to Barlow Twins | Barlow Twins, ResNet-18 · ImageNet-100 | top-1 accuracy: 72.88 -> 76.88 | Helped |
| Add CompMod and MCR to Barlow Twins | Barlow Twins, ResNet-18 · ImageNet, semi-supervised 1% labels | top-1 accuracy: 55.0 -> 56.1 | Helped |
| Add CompMod and MCR to BYOL | BYOL, ImageNet-pretrained network · COCO instance segmentation | AP: 33.1 -> 35.4 | Helped |
| Use all listed augmentations together rather than horizontal flip and rotate only (ablation) | BYOL+, ResNet-18 · CIFAR-10 | linear accuracy: 92.64 -> 93.96 | Helped |
| Vary λ1 and λ2 over the tested values (ablation) | SimCLR+, ResNet-18 · CIFAR-10 | classification accuracy: 91.11 to 93.91 | Mixed |
| Replace representation-space fusion with Mixup at α=0.1 (ablation) | BYOL+ · CIFAR-10 | classification accuracy: 93.96 -> 91.03 | Hurt |
| Fuse features in embedding space instead of representation space (ablation) | BYOL+ · CIFAR-10 | classification accuracy: 93.96 -> 92.58 | Hurt |
| Remove bi-level optimization (ablation) | BYOL+ · CIFAR-10 | classification accuracy: 93.96 -> 92.05 | Hurt |
Limitations
- Theoretical arguments (information-theoretic bounds and a counterfactual theorem) are informal and rely on assumed data-generating processes.
- Hyperparameter sensitivity analysis is limited to CIFAR-10 with SimCLR+, and fusion/optimization ablations are reported only for BYOL+ on CIFAR-10.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 STL-10 Tiny ImageNet ImageNet-100 ImageNet COCO
Benchmarks: CIFAR-10 CIFAR-100 STL-10 Tiny ImageNet ImageNet-100 ImageNet COCO
Metrics: top-1 accuracy top-5 accuracy 5-nn accuracy AP AP50 AP75
Baselines: SimCLR BarlowTwins BYOL SimSiam W-MSE SwAV SSL-HSIC VICReg MoCo CMC
Builds on: SimCLR BYOL Barlow Twins maximum entropy coding MAML-style meta-learning
Closest papers by meaning
- Self-supervised Learning from a Multi-view Perspective (ICLR 2021 · similarity 0.86)
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning (NeurIPS 2025 · similarity 0.85)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.85)
- Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · similarity 0.85)
- Self-Organizing Visual Prototypes for Non-Parametric Representation Learning (ICML 2025 · similarity 0.84)
- Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.