Research map / Label-efficient and robust learning / Recognition and long-tail
Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows analytically and empirically that adding a final-layer ReLU makes supervised contrastive loss learn orthogonal-frame embedding geometry under class imbalance, and proposes batch-binding to guarantee the unique OF minimizer under mini-batch training.
Problem
SCL embedding geometry loses symmetry under class imbalance, prior theory covers only balanced data, and the effect of mini-batch selection on the learned geometry is unclear.
Approach
Refine the unconstrained features model by constraining SCL embeddings to the non-negative orthant (modeling a final ReLU), proving global minimizers form an orthogonal frame irrespective of label distribution, and derive batch-interaction-graph conditions for a unique OF minimizer with batch-binding (adding one example per class to every batch).
Key results
- With ReLU at the final layer, class-mean Gram matrices converge to an orthogonal frame regardless of imbalance ratio (R=1,10,100) on STEP-imbalanced MNIST.
- On CIFAR-100 with ResNet-18, adding ReLU does not hurt NCC test accuracy and improves it in several settings (e.g., LT R=100: 39.71 vs 37.19 without ReLU).
- Batch-binding with batch size 128 on CIFAR-10 gives consistently fast convergence to OF, whereas fixed partitions without binding fail to converge especially under large imbalance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ReLU after the projection head | SCL with ResNet-18 · CIFAR100, balanced | test accuracy: 72.17 $\pm$ 0.23 -> 72.32 $\pm$ 0.60 | No clear effect |
| Add ReLU after the projection head | SCL with ResNet-18 · CIFAR100, 10-Step | test accuracy: 56.58 $\pm$ 0.50 -> 58.16 $\pm$ 0.76 | Helped |
| Add ReLU after the projection head | SCL with ResNet-18 · CIFAR100, 10-LT | test accuracy: 57.10 $\pm$ 1.04 -> 57.88 $\pm$ 1.05 | No clear effect |
| Add ReLU after the projection head | SCL with ResNet-18 · CIFAR100, 100-Step | test accuracy: 43.49 $\pm$ 0.30 -> 43.80 $\pm$ 0.25 | No clear effect |
| Add ReLU after the projection head | SCL with ResNet-18 · CIFAR100, 100-LT | test accuracy: 37.19 $\pm$ 2.50 -> 39.71 $\pm$ 0.09 | Helped |
| Add ReLU at the final layer | SCL representations · Imbalanced data (STEP and long-tailed) | convergence to orthogonal-frame geometry | Helped |
| Use a fixed batch partition without shuffling or batch-binding | SCL training · CIFAR-10, imbalanced training | convergence to orthogonal-frame geometry | Hurt |
| Randomly reshuffle the batch partition every epoch | SCL training · CIFAR-10, imbalanced training | convergence to orthogonal-frame geometry | Helped |
| Apply batch-binding to a fixed batch partition | SCL training · CIFAR-10, imbalanced training | convergence to orthogonal-frame geometry | Helped |
| Apply batch-binding | SCL without ReLU · Balanced training | convergence to ETF geometry | Helped |
Surprising findings
- Without ReLU, SCL geometry changes substantially with imbalance ratio, while adding a single ReLU at the last layer makes it invariant.
- Under mini-batch training, the UFM without non-negativity can have contradictory batch-wise optima, whereas the non-negative constraint lets batch-wise optima be compatible.
Limitations
- Theory covers the unconstrained features model with normalized features and d>=k; it does not model the deep network training dynamics directly.
- Analysis is restricted to the training-time geometry; generalization gains are shown only in limited settings with NCC evaluation.
- Batch-binding guarantees are shown for training geometry; the paper does not study its effect with very large numbers of classes.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 Tiny ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet MNIST
Metrics: distance to OF geometry balanced test accuracy with NCC SCL loss versus theoretical lower bound
Baselines: SCL without ReLU cross-entropy loss MSE loss no batch-shuffling batch-shuffling
Builds on: supervised contrastive loss unconstrained features model neural collapse
Closest papers by meaning
- Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract) (AAAI 2024 · similarity 0.91)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.87)
- Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model (ICML 2024 · similarity 0.87)
- On the Similarities of Embeddings in Contrastive Learning (ICML 2025 · similarity 0.86)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.86)
- Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.