Research map / Learning theory and algorithms / Prediction and decision losses
The Evidence Contraction Issue in Deep Evidential Regression: Discussion and Solution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes evidence contraction in Deep Evidential Regression caused by non-negativity activation constraints on NIG parameters, and proposes a Non-saturating Uncertainty Regularization term to keep gradients nonzero and improve regression error.
Problem
Deep Evidential Regression (DER) has higher prediction error than standard regression; its non-negativity activation constraints on NIG prior parameters cause evidence contraction that stalls optimization of the prediction.
Approach
Theoretically links evidence contraction (virtual observation nu driven to zero under activation functions) to vanishing gradients of the NLL loss w.r.t. the prediction, and adds a regularizer L_U = (y-γ)^2 · ν(α-1)/(β(ν+1)) whose gradient w.r.t. γ is non-zero everywhere to keep optimizing the prediction.
Key results
- On UCI benchmarks the proposed method attains the best or comparable RMSE across datasets, e.g., Power RMSE 3.02 vs 4.23 for Vanilla DER.
- On Davis the method reports MSE 0.26 vs 0.28 for Vanilla DER and ECE 0.14 vs 0.15; on SST-5 MSE improves to 0.52 from 0.54.
- NLL is mixed: it is best on Power (2.58) and Kin8nm (-1.27) but worse than Vanilla DER on Kiba (-1.44 vs -1.54) and Energy/Concrete.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Non-saturating Uncertainty Regularization to DER | DER · UCI regression benchmark | RMSE: Boston: 3.06 ( 0.16) -> 2.67(0.17); Concrete: 5.85 ( 0.15) -> 5.82(0.21) | Mixed |
| Add Non-saturating Uncertainty Regularization to DER | DER · UCI regression benchmark | NLL: Power: 2.81 ( 0.07) -> 2.58(0.01); Concrete: 3.01 ( 0.02) -> 3.11(0.03) | Mixed |
| Add Non-saturating Uncertainty Regularization to DER | DER · Davis | MSE: 0.28(0.00) -> 0.26(0.01) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · Davis | CI: 0.86(0.02) -> 0.87(0.03) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · Davis | ECE: 0.15(0.02) -> 0.14(0.10) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · Davis | NLL: -2.34(0.42) -> -2.37(0.08) | Hurt |
| Add Non-saturating Uncertainty Regularization to DER | DER · Kiba | MSE: 0.19(0.00) -> 0.18(0.01) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · Kiba | ECE: 0.08(0.03) -> 0.05(0.02) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · SST-5 | MSE: 0.54(0.02) -> 0.52(0.01) | Helped |
| Add Non-saturating Uncertainty Regularization to DER | DER · SST-5 | ECE: 0.09(0.02) -> 0.11(0.02) | Hurt |
| Use Exp instead of ReLU activation in DER (ablation) | DER · SST-5 | MSE | Helped |
| Use ReLU activation instead of Exp activation in DER (ablation) | DER · SST-5 | MSE | Hurt |
Surprising findings
- Vanilla DER with ReLU activation performs worst, and the paper argues more severe evidence contraction correlates with poorer MSE, even below standard regression.
Limitations
- Non-saturating term freezes the uncertainty gradient and does not update alpha, beta, nu via the regularizer, a heuristic lower bound rather than a principled fix.
- Mixed NLL results; some datasets (e.g., Kiba, Concrete) show degraded NLL or ECE relative to baselines.
- Gains on several UCI datasets are within reported standard deviations.
Experimental setup
Datasets: UCI regression benchmark (Boston, Concrete, Energy, Kin8nm, Naval, Power, Protein, Wine, Yacht) Davis Kiba SST-5
Benchmarks: UCI regression benchmark Davis Kiba SST-5
Metrics: RMSE NLL MSE CI ECE
Baselines: MC dropout Vanilla DER MT-DER ReLU-DER Exp-DER Standard Regression
Builds on: Deep Evidential Regression multi-task deep evidential regression evidence misdirection regularization
Closest papers by meaning
- Uncertainty Regularized Evidential Regression (AAAI 2024 · similarity 0.90)
- Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? (NeurIPS 2024 · similarity 0.88)
- Uncertainty Estimation by Flexible Evidential Deep Learning (NeurIPS 2025 · similarity 0.87)
- Calibrating LLMs with Information-Theoretic Evidential Deep Learning (ICLR 2025 · similarity 0.86)
- Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods? (ICML 2024 · similarity 0.86)
- Hyper-opinion Evidential Deep Learning for Out-of-Distribution Detection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.