Research map / Architectures and efficiency / RNNs and normalization
Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes implicitly setting the final classifier to the nearest simplex ETF of the penultimate-layer feature means, solved as a Riemannian (Stiefel manifold) optimisation inside a deep declarative node, to speed up neural collapse convergence.
Problem
Training neural networks to neural collapse (simplex ETF classifier) is slow; fixing the classifier to a simplex ETF does not improve convergence speed, and learned classifiers have high run-to-run variance.
Approach
At each iteration, the classifier weights are implicitly set to the simplex ETF nearest the (normalised, centred) feature means, found by solving a proximal Riemannian optimisation over the Stiefel manifold; gradients flow through this inner problem via a deep declarative node using implicit differentiation.
Key results
- On ImageNet ResNet50 at epoch 200, implicit ETF reaches 88.01% train and 65.36% test top-1 versus 77.20%/60.02% for standard training.
- On CIFAR10 VGG13 at epoch 50, implicit ETF gives 95.69% train accuracy vs 93.59% standard and 76.66% fixed ETF, with much smaller across-seed variance.
- Implicit ETF attains a markedly faster drop in CE loss and NC metrics on UFM-10 and remains constant as class count grows to 1000.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the nearest implicit ETF classifier instead of a learnable or fixed classifier | ResNet18 · CIFAR10 test, epoch 50 | top-1 accuracy: 80.47 -> 81.76; 80.63 -> 81.76 | Helped |
| Use the nearest implicit ETF classifier instead of a learnable or fixed classifier | VGG13 · CIFAR10 test, epoch 200 | top-1 accuracy: 90.34 -> 90.98; 72.48 -> 90.98 | Helped |
| Use the nearest implicit ETF classifier instead of a learnable classifier | ResNet50 · CIFAR100 test, epoch 200 | top-1 accuracy: 51.29 -> 50.52 | Hurt |
| Use the nearest implicit ETF classifier instead of a learnable classifier | VGG13 · CIFAR100 test, epoch 200 | top-1 accuracy: 67.54 -> 67.14 | Hurt |
| Use the nearest implicit ETF classifier instead of a fixed ETF classifier | ResNet50 · STL10 test, epoch 200 | top-1 accuracy: 64.33 -> 62.75 | Hurt |
| Use the nearest implicit ETF classifier instead of a learnable classifier | VGG13 · STL10 test, epoch 200 | top-1 accuracy: 81.61 -> 79.94 | Hurt |
| Use the nearest implicit ETF classifier instead of a fixed ETF classifier | ResNet50 · ImageNet test, epoch 200 | top-1 accuracy: 61.47 -> 65.36 | Helped |
| Remove backpropagation through the DDN layer (ablation) | Implicit ETF · small-scale datasets | performance | No clear effect |
| Remove backpropagation through the DDN layer (ablation) | Implicit ETF · feature-level cosine margin | cosine margin | Hurt |
| Use the nearest implicit ETF classifier rather than a fixed ETF or standard classifier | Implicit ETF · real-world classification datasets, convergence over training | convergence speed | Helped |
| Use the nearest implicit ETF classifier | Implicit ETF · real-world classification datasets, repeated runs | performance variability | Helped |
Surprising findings
- Performance remains comparable without backpropagation through the DDN layer on small-scale datasets, making the DDN gradient optional.
- On ImageNet, reported results were generated without the DDN gradient stream due to memory constraints.
Limitations
- DDN backward pass is memory-inefficient and infeasible for large d and C, so ImageNet results omit DDN gradients.
- Gains in convergence are shown mainly over 200 epochs on a few architectures and datasets; the DDN gradient contribution is not clearly established at scale.
- Method relies on stratified batch sampling and EMA of feature means, and the batch-smaller-than-classes case can be unstable early in training.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 ImageNet-1000 Unconstrained Feature Model synthetic data
Benchmarks: CIFAR10 CIFAR100 STL10 ImageNet-1000 UFM synthetic
Metrics: top-1 train accuracy top-1 test accuracy CE loss cosine margin NC1 NC3 W-H equinorm GPU memory forward/backward time
Baselines: Standard learnable classifier Fixed simplex ETF classifier
Builds on: Neural collapse (Papyan et al.) Deep declarative networks Fixed simplex ETF classifier Automatic Gradient Descent (AGD)
Closest papers by meaning
- Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Model (ICML 2024 · similarity 0.88)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.88)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.88)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.88)
- Unlocking Better Closed-Set Alignment Based on Neural Collapse for Open-Set Recognition (AAAI 2025 · similarity 0.87)
- Neural Collapse To Multiple Centers For Imbalanced Data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.