Research map / Label-efficient and robust learning / Self-supervised and contrastive
Contrastive Self-Supervised Learning As Neural Manifold Packing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLAMP, a self-supervised method that treats each image's augmentations as an ellipsoidal sub-manifold and trains with a short-range repulsive packing loss, reaching competitive linear-eval accuracy on ImageNet and CIFAR-10.
Problem
Contrastive self-supervised learning relies on pointwise pairwise comparisons, and its geometric foundations (how class manifolds are arranged for separability) are underexplored; the paper asks whether manifold packing can serve as a representation learning objective.
Approach
Each image's augmented views form an ellipsoidal augmentation sub-manifold, and the loss is a log of a short-range repulsive overlap energy between sub-manifold pairs (sizes from covariance trace, centers from view means), with no alignment/uniformity terms and no negatives beyond neighboring sub-manifolds.
Key results
- Linear evaluation on ImageNet-100 reaches 85.12% (4 views), above MMCR's 82.88% and the other listed baselines, claimed as new SOTA on that benchmark.
- On ImageNet-1K linear evaluation with 100-epoch pretraining, CLAMP reaches 69.50% (4 views) and 70.04% (8 views), below MMCR's 71.5% and SwAV's 72.1%.
- CIFAR-10 linear evaluation reaches 90.21% top-1, below BYOL (92.58%) and Barlow Twins (92.10%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MMCR with CLAMP (4 views) | ImageNet-100 linear evaluation · ImageNet-100 | accuracy: 82.88 -> 85.12 $ \pm .05$ | Helped |
| Replace SwAV with CLAMP (4 views) | ImageNet-1K linear evaluation · ImageNet-1K | accuracy: 72.1 -> 69.50 $\pm .14$ | Hurt |
| Replace SwAV with CLAMP (4 views) | ImageNet-1K 1% semi-supervised evaluation · ImageNet-1K 1% split | accuracy: 49.8 -> 47.38 $\pm .56$ | Hurt |
| Replace SwAV with CLAMP (4 views) | ImageNet-1K 10% semi-supervised evaluation · ImageNet-1K 10% split | accuracy: 66.9 -> 65.10 $\pm .30$ | Hurt |
| Use CLAMP instead of BYOL | VOC object detection with Faster R-CNN R50-C4 · VOC2007 test | mAP: 55.6 -> 55.7 | Helped |
| Use CLAMP instead of BYOL | VOC object detection with Faster R-CNN R50-C4 · VOC2007 test | AP50: 82.3 -> 82.3 | No clear effect |
| Use CLAMP instead of BYOL | VOC object detection with Faster R-CNN R50-C4 · VOC2007 test | AP75: 62.0 -> 62.4 | Helped |
| Increase the number of augmentation views from 2 to 4 (ablation) | CLAMP · ImageNet-1K 30% linear evaluation | top 1 accuracy: 55.17 -> 58.29 | Helped |
| Increase the number of augmentation views from 4 to 8 (ablation) | CLAMP · ImageNet-1K 30% linear evaluation | top 1 accuracy: 58.29 -> 58.95 | Helped |
| Vary batch size from 256 to 1024 (ablation) | CLAMP with 4 views · ImageNet-1K linear evaluation | top 1 accuracy: 69.02; 69.43; 69.11 | No clear effect |
Surprising findings
- Linear accuracy depends only weakly on batch size (256 to 1024 gives 69.02 to 69.11 top-1 on a 4-view ablation).
Limitations
- Brain-Score linear mapping may miss nonlinear model-brain relations, and CLAMP does not claim the same computations as cortex.
- Backpropagation with large-batch contrastive optimization is not biologically realistic.
- Orientation information is omitted because estimating it needs about 100 augmentations per input, which is infeasible at CIFAR-10 or ImageNet scale.
Experimental setup
Datasets: ImageNet-1K ImageNet-100 CIFAR-10 MNIST VOC2007+2012 Brain-Score neural datasets (FreemanZiemba2013, MajajHong2015)
Benchmarks: ImageNet-100 linear evaluation ImageNet-1K linear evaluation ImageNet-1K semi-supervised 1%/10% CIFAR-10 linear evaluation VOC object detection Brain-Score V1/V2/V4/IT
Metrics: top-1 accuracy mAP AP50 AP75 Brain-Score
Baselines: SimCLR SwAV Barlow Twins BYOL VICReg CorInfoMax MoCo-V2 SimSiam MMCR A&U DeepCluster V2
Builds on: MMCR Alignment-Uniformity SimCLR LARS solo-learn
Closest papers by meaning
- Contrastive-Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT (NeurIPS 2024 · similarity 0.85)
- Exploring the Enigma of Neural Dynamics Through A Scattering-Transform Mixer Landscape for Riemannian Manifold (ICML 2024 · similarity 0.84)
- Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.84)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.84)
- Data-free Neural Representation Compression with Riemannian Neural Dynamics (ICML 2024 · similarity 0.83)
- Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.