Research map / Label-efficient and robust learning / Self-supervised and contrastive
How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes how labeling error from strong augmentations (e.g., RRC) affects contrastive learning's downstream classification risk and error, and proposes truncated SVD on raw data (plus moderate embedding dimension, data inflation, weak augmentation) to reduce labeling error while noting SVD can reduce augmentation-graph connectivity.
Problem
Contrastive learning theory typically assumes label consistency between augmented views of the same sample; strong augmentations like random resized crop break this, creating labeling error that degrades downstream classification.
Approach
Derives bounds on downstream classification risk that explicitly include intra-class and inter-class (false positive) augmentation overlap terms under labeling error, and proposes truncated SVD applied to original samples to discard semantically irrelevant components and thereby reduce labeling error, while analyzing the trade-off with augmentation-graph connectivity.
Key results
- On CIFAR-10 with ResNet-18 and the RRC/color jitter/flip/grayscale/blur augmentation set, truncated SVD with q=20 raised linear-probe accuracy from 68.82% to 69.87%, and q=30 gave 69.48%.
- Under the weak augmentation of prior work, SVD (q=25) combined with data inflation raised CIFAR-10 ResNet-18 accuracy from 71.54% to 72.55%.
- Across augmentation sets in the SVD-augmentation table, truncated SVD with q=30 improved accuracy by an average of about 0.97%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply truncated SVD at a selected q (ablation) | SimCLR with ResNet-18 · CIFAR-10, T1 | top-1 accuracy (%): 68.82 -> 69.87 | Helped |
| Apply truncated SVD at a selected q (ablation) | SimCLR with ResNet-50 · CIFAR-10, T1 | top-1 accuracy (%): 63.20 -> 63.96 | Helped |
| Apply truncated SVD at a selected q (ablation) | SimCLR · CIFAR-10, RRC; CIFAR-100 and STL-10, T1 | top-1 accuracy (%): 58.56 -> 58.83; 38.48 -> 40.10; 71.54 -> 73.12 | Helped |
| Apply SVD (q=30) (ablation) | SimCLR with ResNet-18 · CIFAR-10, T2–T7 and RRC(0.08,0.5) | top-1 accuracy (%): 62.90 -> 64.86; 50.53 -> 51.00; 60.00 -> 61.57; 56.67 -> 57.85; 54.97 -> 55.69; 54.09 -> 54.75; 57.11 -> 58.10 | Helped |
| Vary the truncation parameter q (ablation) | SimCLR · CIFAR-10, CIFAR-100 and STL-10 | top-1 accuracy (%): CIFAR-10 ResNet-18: 69.75 at q=25, 69.87 at q=20, 68.26 at q=10; CIFAR-100 ResNet-18: 40.10 at q=25, 38.10 at q=10 | Mixed |
| Set embedding dimension to a moderate value (k=512 or k=1024) (ablation) | SimCLR with truncated SVD · CIFAR-10, CIFAR-100 and STL-10 | top-1 accuracy (%): ResNet-18 CIFAR-10: 67.71 (k=128) -> 69.09 (k=1024); ResNet-18 CIFAR-100: 35.00 (k=128) -> 37.78 (k=1024); ResNet-18 STL | Mixed |
| Add data inflation (ablation) | SimCLR with SVD (q=25), spectral loss · CIFAR-10, T8 | top-1 accuracy (%): 71.64 -> 72.55 | Helped |
| Add data inflation (ablation) | SimCLR with SVD (q=25), InfoNCE loss · CIFAR-10, T8 | top-1 accuracy (%): 70.92 -> 71.21 | Helped |
Surprising findings
- Discarding the two smallest singular values of raw images improves accuracy, but SVD can reduce augmentation-graph connectivity and hurt performance when the embedding dimension is too small or too large.
- Overly large embedding dimension can degrade downstream accuracy, contrary to the theoretical expectation that larger k increases graph connectivity.
Limitations
- The optimal truncation parameter q* is unknown and must be chosen heuristically.
- Empirical observation that overly large embedding dimension hurts performance is not explained by the theory.
- Experiments are limited to small image benchmarks (CIFAR, STL-10) with ResNet backbones.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 STL-10
Benchmarks: CIFAR-10 CIFAR-100 STL-10
Metrics: top-1 linear probe accuracy downstream classification risk bounds
Baselines: SimCLR without SVD data inflation strategy (prior work)
Builds on: SimCLR spectral contrastive loss HaoChen et al. augmentation graph analysis data inflation with DDPM
Closest papers by meaning
- An Augmentation-Aware Theory for Self-Supervised Contrastive Learning (ICML 2025 · similarity 0.89)
- Investigating Why Contrastive Learning Benefits Robustness against Label Noise (ICML 2022 · similarity 0.89)
- Rethinking Weak Supervision in Helping Contrastive Learning (ICML 2023 · similarity 0.88)
- An Investigation of Representation and Allocation Harms in Contrastive Learning (ICLR 2024 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Perfect Alignment May be Poisonous to Graph Contrastive Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.