Research map / Optimization / Optimizers
A PID Controller Approach for Adaptive Probability-dependent Gradient Decay in Model Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PID-AGD, which uses a PID controller on a validation-set relative calibration error to adapt a probability-dependent gradient decay coefficient β in the softmax loss, plus a learning-rate compensation rule, to improve calibration without sacrificing accuracy.
Problem
Deep classifiers are overconfident, and calibration tends to overfit earlier than accuracy during cross-entropy training, so accuracy and calibration are not optimized consistently.
Approach
Treat the gradient decay coefficient β of a parametric softmax loss as a controlled variable, with validation-set relative calibration error (RCE) as the feedback signal for a PID controller that adjusts β each epoch; a derived learning-rate rule compensates for the resulting gradient-magnitude changes.
Key results
- On CIFAR-100 ResNet18, ECE drops to 0.006 for the proposed method versus 0.160 uncalibrated and 0.033 for temperature scaling (Table 1).
- On CIFAR-100 ResNet35 with the PID controller and gradient compensation, accuracy is 74.7% versus 73.8% for SGD softmax baseline, with ECE 0.012 versus 0.172 (Table 3).
- On 102 Flower ResNet50, ECE is 0.033 versus 0.075 for the next best method (TS-AvUC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PID-based adaptive probability-dependent gradient decay | ResNet18 · CIFAR-10 | ECE: 0.041±0.010 -> 0.007±0.002 | Helped |
| apply PID-based adaptive probability-dependent gradient decay | ResNet35 · CIFAR-100 | accuracy: 73.8±0.30 -> 75.4±0.39 | Helped |
| apply PID-based adaptive probability-dependent gradient decay | ResNet35 · CIFAR-100 | ECE: 0.172±0.022 -> 0.011±0.003 | Helped |
| add gradient compensation to PID-controlled SGD (ablation) | ResNet35 · CIFAR-100 | accuracy: 72.5% -> 74.7% | Helped |
| add gradient compensation to PID-controlled SGD (ablation) | ResNet35 · CIFAR-100 | ECE: 0.022 -> 0.012 | Helped |
| replace SGD with Adam under PID control (ablation) | ResNet35 · CIFAR-100 | accuracy: 73.8% -> 63.5% | Hurt |
| increase gradient decay factor β from 0.1 to 20 (ablation) | ResNet34 · Tiny-ImageNet | ECE: 0.224 -> 0.015 | Helped |
| increase gradient decay factor β from 0.1 to 20 (ablation) | ResNet34 · Tiny-ImageNet | Top-1 Acc (%): 53.7 -> 52.8 | Mixed |
| increase gradient decay factor β from 0.01 to 20 (ablation) | ResNet50 · Tiny-ImageNet | ECE: 0.249 -> 0.045 | Helped |
| increase gradient decay factor β from 0.01 to 20 (ablation) | ResNet50 · Tiny-ImageNet | Top-1 Acc (%): 56.2 -> 55.9 | Mixed |
| vary PID controller P, I, and D settings (ablation) | ResNet35 · CIFAR-100 | accuracy | No clear effect |
| use excessively high PID parameter settings (ablation) | ResNet35 · CIFAR-100 | ECE stability | Hurt |
Surprising findings
- Adam optimizer with the PID-controlled dynamic objective drops CIFAR-100 ResNet35 accuracy to 63.5%, far below the 73.8% baseline.
- Post-hoc temperature scaling and vector scaling are not always worse; on some MCE metrics the proposed method is worse than the baselines (e.g., CIFAR-100 ResNet18 MCE 0.068 vs 0.052 for TS-AvUC).
Limitations
- No theoretical analysis of how gradient decay affects confidence distribution.
- Calibration responds slowly to β changes, implying a time lag that the PID controller handles only empirically.
- The learning-rate compensation assumes uniform sample probabilities in [0,1], which may not hold during training.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN 102 Flower Tiny-ImageNet VisDrone COCO
Benchmarks: CIFAR-10 CIFAR-100 SVHN 102 Flower Tiny-ImageNet VisDrone COCO
Metrics: ECE MCE AdaECE accuracy
Baselines: Histogram Binning Temperature Scaling Vector Scaling TS-AvUC CosFace Center loss DCA Softmax cross-entropy
Builds on: temperature scaling DCA label smoothing
Closest papers by meaning
- Uncertainty Weighted Gradients for Model Calibration (CVPR 2025 · similarity 0.84)
- An Effective Dynamic Gradient Calibration Method for Continual Learning (ICML 2024 · similarity 0.84)
- A Benchmark Study on Calibration (ICLR 2024 · similarity 0.84)
- On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization (ICML 2024 · similarity 0.84)
- Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer (ICML 2024 · similarity 0.84)
- Calibration Bottleneck: Over-compressed Representations are Less Calibratable (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.