Research map / Label-efficient and robust learning / Noisy labels
Optimized Gradient Clipping for Noisy Label Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Optimized Gradient Clipping (OGC), which adapts the probability-gradient clipping threshold each step by fitting a 2-GMM to cross-entropy values and solving for the threshold that yields a target noise-to-clean gradient ratio, applied to CE/FL and GCE for noisy-label learning.
Problem
Fixed gradient-clipping thresholds on model-predicted probabilities, tuned on validation data, ignore how gradient distributions from clean and noisy samples shift during training, limiting robustness to label noise.
Approach
Dynamically set the clipping threshold at each training step by modeling clean and noisy cross-entropy distributions with a 2-component GMM and choosing the smallest threshold (via binary search) that keeps the estimated post-clipping noise-to-clean gradient ratio at or above a predefined value 1+epsilon, so that noisy gradients are bounded while clean fitting is preserved.
Key results
- On CIFAR-10 the OGC-based CE+OGC achieves 76.99% average accuracy and FL+OGC 77.10%, the top-2 averages across noise settings, versus 69.88% for GCE and 75.89% for SCE.
- On CIFAR-10 with real-world noise (CIFAR-10N Worst), CE+OGC improves over CE by 17.02 points.
- Optimized threshold (74.03 average) beats fixed (60.36), linear (68.44), and EMA (68.57) threshold schedules on CIFAR-10; OGC applied to GCE raises its average from 65.61 to 72.95.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply optimized dynamic thresholding instead of a fixed, linear, or EMA threshold | Gradient clipping on CIFAR-10 · CIFAR-10, noise-type average | test accuracy (%): 74.03 vs 68.57 | Helped |
| Apply OGC to GCE | GCE on CIFAR-10 · CIFAR-10, multiple noise types | test accuracy (%): + 0.87; + 3.96; + 21.01; + 10.46; + 0.42 | Helped |
| Apply OGC to CE | CE on CIFAR-10 and CIFAR-100 · CIFAR-10 and CIFAR-100, multiple noise types | test accuracy (%): +17.02% | Helped |
| Apply OGC to CE or FL | CE and FL · CIFAR-10 and CIFAR-100, noisy-label settings | test accuracy (%) | Mixed |
| Use a smaller ε, such as ε = 0.01 (ablation) | OGC gradient-ratio constraint · CIFAR-10, Sym-50% and Sym-80% | test accuracy | Mixed |
| Use a larger ε, such as ε = 5.0 (ablation) | OGC gradient-ratio constraint · CIFAR-10, Sym-50% and Sym-80% | test accuracy | Hurt |
| Vary the FIFO queue size q (ablation) | OGC 2-GMM estimation · CIFAR-10, Sym-80% | test accuracy | No clear effect |
| Increase the threshold-update time frame s beyond a certain threshold (ablation) | OGC threshold updates · CIFAR-10, Sym-80% | test accuracy | Hurt |
Surprising findings
- Non-robust losses such as CE and FL become among the top performers when combined with OGC, despite being among the least robust losses without it.
Limitations
- The 2-GMM may model clean and noisy distributions inaccurately when the noise distribution is complex or the noise rate is high.
- Requires tuning of epsilon_0 (the single hyper-parameter), with the best value varying across datasets and noise settings.
- The theoretical bounds assume the loss depends only on the predicted probability of the given label and rely on assumptions such as clean samples having lower cross-entropy than noisy ones.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10N CIFAR-100N WebVision 1.0 ILSVRC 2012
Benchmarks: CIFAR-10 CIFAR-100 CIFAR-10N CIFAR-100N WebVision 1.0 ILSVRC 2012
Metrics: top-1 test accuracy
Baselines: CE FL MAE GCE SCE PHuber-CE Taylor-CE NCE+RCE JS LC-CE NCE+NNCE
Builds on: PHuber-CE LogitClip Generalized Cross Entropy Focal Loss
Closest papers by meaning
- Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.86)
- To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.84)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.84)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.84)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.84)
- Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.