Research map / Label-efficient and robust learning / Recognition and long-tail
Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes fixed (non-trainable) class prototypes added to the batch in supervised-contrastive loss to engineer the embedding geometry, and shows that in the many-prototypes limit SCL reduces to cross-entropy with a fixed classifier and normalized embeddings.
Problem
Prior SCL methods for imbalanced classification use trainable prototypes (PaCo, BCL), so the isolated effect of prototypes on the implicit feature geometry (neural collapse / ETF) is unclear.
Approach
Replace trainable class centers with fixed prototypes that form a prescribed reference geometry, appended to each batch in the SCL loss; in the limit of many prototypes the loss becomes CE with a fixed classifier and normalized features.
Key results
- Gram matrices of SCL with 100 fixed prototypes per class show a more symmetric, ETF-like geometry than vanilla SCL or BCL on STEP-imbalanced CIFAR-10 with ResNet-18 at epoch 350.
- Theoretically, when prototype count far exceeds batch size, batch-wise SCL loss converges to a CE-like loss with fixed classifier and normalized embeddings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fixed prototypes with an engineered geometry to SCL | SCL · STEP imbalanced CIFAR-10 | feature-geometry symmetry | Helped |
Limitations
- Evidence is limited to a short student abstract with qualitative Gram matrix comparisons; no systematic test accuracy results shown in the text.
- Theory is restricted to the limit where the number of prototypes far exceeds the batch size.
- Effect of prototype geometry on minority-class generalization is left unexplored.
Experimental setup
Datasets: CIFAR-10
Benchmarks: STEP-imbalanced CIFAR-10
Metrics: Gram matrix visualization test accuracy
Baselines: vanilla SCL BCL
Builds on: supervised contrastive loss balanced contrastive learning PaCo neural collapse
Closest papers by meaning
- Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching (ICLR 2024 · similarity 0.91)
- Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses (ICML 2024 · similarity 0.86)
- BCE vs. CE in Deep Feature Learning (ICML 2025 · similarity 0.86)
- Decoupled Contrastive Learning for Long-Tailed Recognition (AAAI 2024 · similarity 0.86)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.86)
- Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.