Research map / Label-efficient and robust learning / Recognition and long-tail
Non-negative Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Non-negative Contrastive Learning (NCL), which adds a ReLU non-negativity reparameterization to contrastive objectives so that learned features are equivalent to a symmetric NMF of the augmentation co-occurrence matrix, yielding sparser, more disentangled, interpretable features.
Problem
Contrastive learning features are dense, rotation-symmetric and non-interpretable, so individual feature dimensions do not correspond to semantic concepts.
Approach
Impose non-negativity on contrastive encoder outputs (e.g., via ReLU), which the paper shows is mathematically equivalent to symmetric NMF of the normalized co-occurrence matrix; this breaks rotation symmetry and yields sparse, class-consistent, orthogonal features with identifiability and Bayes-optimal linear-probe guarantees.
Key results
- NCL improves ImageNet-100 linear probing from 68.7 to 69.4 and fine-tuning from 77.3 to 79.2 over CL (SimCLR).
- On ImageNet-100 feature disentanglement, SEPIN@100 rises from 0.69 for CL to 3.87 for NCL (x10^-2).
- Feature selection to 512 of 2048 dimensions with EA keeps NCL linear probing at 68.9 (matches all-feature), while CL drops to 66.2 with random selection; around 90% of NCL feature dimensions are zero per sample.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add non-negative feature outputs | SimCLR · CIFAR-100 | LP accuracy: 58.6 ± 0.2 -> 59.7 ± 0.4 | Helped |
| add non-negative feature outputs | SimCLR · ImageNet-100 | FT accuracy: 77.3 ± 0.5 -> 79.2 ± 0.4 | Helped |
| add non-negative feature outputs | SimCLR · ImageNet-C (100 classes) | LP accuracy: 34.5 ± 0.2 -> 36.1 ± 0.3 | Helped |
| add non-negative feature outputs | SimCLR · ImageNet-100, feature selection using all 2048 dimensions | mAP@10: 10.9 ± 0.2 -> 14.2 ± 0.2 | Helped |
| select the top 512 dimensions by expected activation with ReLU (ablation) | NCL · ImageNet-100 | mAP@10: 14.2 ± 0.2 -> 14.2 ± 0.2 | No clear effect |
| select the top 512 dimensions by expected activation with ReLU (ablation) | NCL · ImageNet-100 | transfer accuracy: 19.9 ± 0.1 -> 19.8 ± 0.1 | No clear effect |
| add non-negative feature outputs | SimCLR · ImageNet-100 | SEPIN@100: 0.69 ± 0.01 -> 3.87 ± 0.04 | Helped |
| add non-negative feature outputs | SimCLR · ImageNet-100 | SEPIN@all: 0.47 ± 0.01 -> 0.48 ± 0.01 | No clear effect |
| add non-negative feature outputs | SCL · CIFAR-10 | linear accuracy (encoder): 88.0% -> 88.6% | Helped |
| replace non-negative feature outputs with L1 regularization | SCL · CIFAR-10 | linear accuracy (encoder): 88.0 -> 87.2 | Hurt |
| use the ReLU-forward, GELU-gradient reparameterization to revive dead neurons (ablation) | NCL · CIFAR-10 and CIFAR-100 | feature diversity | Helped |
| use the ReLU-forward, GELU-gradient reparameterization to revive dead neurons (ablation) | NCL · ImageNet-100 | performance relative to ReLU | Hurt |
Surprising findings
- A ReLU-based non-negativity constraint on projector outputs gives comparable downstream accuracy to unconstrained CL on CIFAR-10/100 while greatly improving interpretability.
- A naive l1 sparsity regularizer yields almost no sparsity (0.03%) while NCL reaches about 69.65% sparsity with higher accuracy.
Limitations
- Theoretical guarantees rely on latent class assumptions (positive generation, small class overlap, existence of unique samples) that are only approximately verified on real data.
- Dead-neuron fix via GELU-gradient trick works on CIFAR but is inferior to ReLU on ImageNet-100; systematic study left open.
- Disentanglement on Dsprites gives modest MIG (0.065) and the default augmentations may be suboptimal for toy data.
Experimental setup
Datasets: ImageNet-100 CIFAR-10 CIFAR-100 Dsprites ImageNet-C ImageNet-Sketch Stylized ImageNet
Benchmarks: ImageNet-100 CIFAR-10 CIFAR-100 Stylized ImageNet ImageNet-C ImageNet-Sketch Dsprites
Metrics: linear probing accuracy fine-tuning accuracy mAP@10 SEPIN@k MIG class consistency rate feature sparsity
Baselines: CL (SimCLR / InfoNCE) spectral contrastive learning l1-regularized contrastive learning SupCon CE
Builds on: SimCLR spectral contrastive loss InfoNCE non-negative matrix factorization SupCon
Closest papers by meaning
- An Empirical Study on Disentanglement of Negative-free Contrastive Learning (NeurIPS 2022 · similarity 0.87)
- Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning (ICLR 2022 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework (ICML 2022 · similarity 0.87)
- Zero-Mean Regularized Spectral Contrastive Learning: Implicitly Mitigating Wrong Connections in Positive-Pair Graphs (ICLR 2024 · similarity 0.87)
- Generalization Analysis for Supervised Contrastive Representation Learning under Non-IID Settings (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.