Research map / Optimization / Optimizers
Soft ascent-descent as a stable and flexible alternative to flooding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SoftAD, a smooth pointwise soft ascent-descent variant of flooding (rho(x)=sqrt(x^2+1)-1) that penalizes both over- and under-performing losses, yielding smaller generalization gaps and model norms than flooding and SAM at no extra gradient cost.
Problem
Flooding sets a fixed loss threshold and switches between gradient descent and ascent on the mini-batch average loss, but threshold choice is hard and it only asks the mean loss to sit near the threshold, which may not control complexity or test loss.
Approach
Apply the ascent/descent switch pointwise to each loss through a smooth bounded truncation phi = rho' with rho a pseudo-Huber-type function, so borderline points get less weight and the loss distribution is pushed to concentrate around the threshold, with the same per-step cost as vanilla GD.
Key results
- SoftAD gives the smallest test-minus-train loss gap on all seven datasets, e.g. 1.168 vs 1.239 (Flood) on CIFAR-10 and 0.004 vs 0.011 on the Gaussian simulation.
- SoftAD attains the smallest model norm across all datasets and models tested, without any explicit regularization.
- Test accuracy of all four methods (ERM, Flooding, SAM, SoftAD) is described as comparable, with SAM slightly ahead, while SAM costs double the gradient computation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SoftAD instead of ERM | classification models · Gaussian, Sinusoid, Spiral, CIFAR-10, CIFAR-100, FashionMNIST, SVHN | loss generalization gap: 0.080 -> 0.004; 0.150 -> 0.016; 0.297 -> 0.087; 3.265 -> 1.168; 7.603 -> 2.701; 0.801 -> 0.422; 0.762 -> 0.362 | Helped |
| Use SoftAD instead of Flood | classification models · Gaussian, Sinusoid, Spiral, CIFAR-10, CIFAR-100, FashionMNIST, SVHN | loss generalization gap: 0.011 -> 0.004; 0.058 -> 0.016; 0.119 -> 0.087; 1.239 -> 1.168; 3.114 -> 2.701; 0.436 -> 0.422; 0.436 -> 0.362 | Helped |
| Use SoftAD instead of SAM | classification models · Gaussian, Sinusoid, Spiral, CIFAR-10, CIFAR-100, FashionMNIST, SVHN | loss generalization gap: 0.024 -> 0.004; 0.096 -> 0.016; 0.154 -> 0.087; 1.512 -> 1.168; 3.672 -> 2.701; 0.639 -> 0.422; 0.493 -> 0.362 | Helped |
| Use SoftAD instead of ERM, Flooding, or SAM | image classification models · CIFAR-10, CIFAR-100, FashionMNIST, SVHN | test loss | Helped |
| Use SAM instead of the other compared methods | image classification models · CIFAR-10, CIFAR-100, FashionMNIST, SVHN | test accuracy | Mixed |
| Use SoftAD instead of ERM, Flooding, or SAM | classification models · all datasets and models tested | model norm | Helped |
Surprising findings
- The validation-selected threshold theta for SoftAD tends to be larger than that for Flooding, uniformly across datasets, yet SoftAD still achieves smaller test loss.
Limitations
- How to set the threshold theta remains unanswered and requires user judgement.
- Theoretical understanding of the implicit regularization effect is incomplete.
- Experiments are limited to a narrow set of loss functions and mostly cross-entropy.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 FashionMNIST SVHN synthetic 2D Gaussian/sinusoid/spiral datasets
Benchmarks: CIFAR-10 CIFAR-100 FashionMNIST SVHN synthetic 2D classification
Metrics: test accuracy test cross-entropy loss loss generalization gap model norm
Baselines: ERM Flooding SAM iFlood AdaFlood
Builds on: flooding (Ishida et al. 2020) sharpness-aware minimization gradient norm penalization iFlood AdaFlood
Closest papers by meaning
- Do We Need Zero Training Loss After Achieving Zero Training Error? (ICML 2020 · similarity 0.85)
- Optimal Regularization can Mitigate Double Descent (ICLR 2021 · similarity 0.85)
- A Universal Growth Rate for Learning with Smooth Surrogate Losses (NeurIPS 2024 · similarity 0.84)
- Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer (NeurIPS 2024 · similarity 0.84)
- Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains (AAAI 2024 · similarity 0.83)
- Incorporating Surrogate Gradient Norm to Improve Offline Optimization Techniques (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.