Research map / Architectures and efficiency / RNNs and normalization
Improving Generalization of Deep Neural Networks by Optimum Shifting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes optimum shifting (OS), a penalty-free method that moves final-layer weights within the under-determined solution space of a fixed-input/fixed-output linear system to reduce Hessian trace without changing training loss, applied stochastically on small batches justified by Neural Collapse.
Problem
Sharp minima in deep networks are associated with poorer generalization, and existing sharpness minimization methods are penalty-based; the paper asks whether a trained network can be moved to a flatter point with the same training loss.
Approach
Treat the linear layer's matrix multiplication A V = Z with fixed inputs and outputs as an under-determined system and replace V with its minimum-Frobenius-norm solution V* = A^T(AA^T)^{-1}Z, which the authors argue bounds the Hessian trace; stochastic OS applies this on small batches (e.g., 300 samples) so that Neural Collapse makes the loss change negligible over the full dataset.
Key results
- On CIFAR-100 with ResNet-50 and mixup, SAM+SOS improves test accuracy from 81.65% to 82.41%.
- Applying SOS to pretrained ResNet-50 on ImageNet raises top-1 accuracy from 76.14% to 76.61% and top-5 from 92.86% to 93.14%.
- Gains are generally small (typically under 1 point), with SOS improving SGD and SAM baselines across VGG-16, ResNet-18/50 and DenseNet-201 on CIFAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SOS to a trained VGG-16 | VGG-16 · CIFAR-100 Basic | test accuracy: 72.12 ± 0.52 -> 72.23 ± 0.54 | No clear effect |
| Apply SOS to a trained VGG-16 | VGG-16 · CIFAR-100 Mixup | test accuracy: 73.22 ± 0.41 -> 73.38 ± 0.81 | No clear effect |
| Apply SOS to a trained VGG-16 | VGG-16 · CIFAR-10 Basic | test accuracy: 92.01 ± 0.45 -> 92.28 ± 0.22 | No clear effect |
| Apply SOS to a trained VGG-16 | VGG-16 · CIFAR-10 Mixup | test accuracy: 93.21 ± 0.32 -> 93.35 ± 0.43 | No clear effect |
| Apply SOS to a pretrained ResNet50 | ResNet50 · ImageNet | top-1 accuracy: 76.14 -> 76.61 | Helped |
| Apply SOS to a pretrained ResNet50 | ResNet50 · ImageNet | top-5 accuracy: 92.86 -> 93.14 | Helped |
| Apply SOS during SGD training | VGG-16 · CIFAR-100 Basic | test accuracy: 72.12 ± 0.52 -> 73.35 ± 0.43 | Helped |
| Apply SOS during SAM training | VGG-16 · CIFAR-100 Basic | test accuracy: 74.63 ± 0.12 -> 74.92 ± 0.03 | Helped |
| Set the SOS batch size to 300 rather than 200 (ablation) | VGG-16 · CIFAR-100 | test accuracy: 72.91 ± 0.19 -> 73.35 ± 0.43 | Helped |
| Set the SOS batch size to 400 rather than 300 (ablation) | VGG-16 · CIFAR-100 | test accuracy: 73.35 ± 0.43 -> 73.18 ± 0.43 | Hurt |
Surprising findings
- Applying SOS on the full training set is not under-determined for a 1024x100 last layer, so the method only works with small batches, with batch size acting as a sensitive hyperparameter (best at 300 on CIFAR-100).
Limitations
- Relies on Neural Collapse, so the authors say it is currently limited to image tasks and not NLP.
- Stochastic batch size is a sensitive hyperparameter trading off loss preservation against degrees of freedom.
- Reported improvements are small and often within reported standard deviations.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet PASCAL VOC
Benchmarks: CIFAR-10 CIFAR-100 ImageNet PASCAL VOC
Metrics: test accuracy top-1 accuracy top-5 accuracy mAP Hessian trace top-1 Hessian eigenvalue
Baselines: SGD with weight decay SAM
Builds on: sharpness-aware minimization neural collapse mixup
Closest papers by meaning
- Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning (ICML 2022 · similarity 0.88)
- Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification (NeurIPS 2025 · similarity 0.86)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.86)
- Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss (ICML 2024 · similarity 0.86)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.