Research map / Architectures and efficiency / RNNs and normalization
Fix your classifier: the marginal value of training the last weight layer
Elad Hoffer, Itay Hubara, Daniel Soudry
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fix the classifier weights instead of learning them | ResNet56 · Cifar10 validation | accuracy: 93.03% -> 93.14% | No clear effect |
| Fix the classifier weights instead of learning them | DenseNet(k=12) · Cifar100 validation | accuracy: 77.73% -> 77.67% | No clear effect |
| Fix the classifier weights instead of learning them | ResNet50 · ImageNet validation | accuracy: 75.3% -> 75.3% | No clear effect |
| Fix the classifier weights instead of learning them | DenseNet169 · ImageNet validation | accuracy: 76.2% -> 76% | No clear effect |
| Fix the classifier weights instead of learning them | ShuffleNet · ImageNet validation | accuracy: 65.9% -> 65.4% | Hurt |
| Replace learned embedding and classifier weights with fixed word2vec embeddings | 2-layer LSTM (h=512) · WikiText-2 validation | perplexity: 74.1 -> 81.2 | Hurt |
| Use a random orthogonal transform instead of learned or pretrained word embeddings (ablation) | Language model · WikiText-2 | perplexity | Hurt |
| Use fixed word2vec embeddings rather than a random orthogonal transform (ablation) | Language model · WikiText-2 | perplexity | Improved |
| Fix the softmax scale at α=1 or α=10 instead of learning it (ablation) | ResNet56 · Cifar10 validation | accuracy | No clear effect |
| Replace softmax with the cosine angle loss and discard the scale parameter (ablation) | Fixed-classifier models · Classification validation | accuracy | Hurt |
Closest papers by meaning
- Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood (NeurIPS 2024 · similarity 0.84)
- Neural Collapse in Cumulative Link Models for Ordinal Regression: An Analysis with Unconstrained Feature Model (NeurIPS 2025 · similarity 0.84)
- Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining (NeurIPS 2025 · similarity 0.84)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.84)
- Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks (NeurIPS 2025 · similarity 0.83)
- How many classifiers do we need? (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
