Research map / Learning theory and algorithms / Prediction and decision losses
Set-Valued Sensitivity Analysis of Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a set-valued (Lipschitz-like) sensitivity analysis of DNN training solutions w.r.t. training data perturbations, proving the property for deep fully connected ReLU networks and using graphical derivatives to estimate perturbed solution sets without retraining (e.g., ResNet56 on CIFAR-10).
Problem
Sensitivity of DNN training solutions (weight sets) to perturbations in training inputs, where the solution set may be non-unique and the Hessian may be singular, so classical influence-function/implicit-function analysis fails.
Approach
Treats the first-order optimality solution mapping as set-valued and analyzes it with Lipschitz-like property and graphical derivatives, yielding set-to-set sensitivity bounds and a Hessian-pseudoinverse estimate of perturbed solution sets that does not require non-singular Hessians.
Key results
- On ResNet56/CIFAR-10, the estimated solution reduces loss on perturbed points from 0.0041 (original) to 1.9e-5 for one perturbed point.
- Relative difference between original and estimated solution is 19e-5 versus 5.37e-5 for retrained solution.
- Toy two-layer linear network gives Lipschitz modulus 0.2 and estimated solutions reduce risk from 0.4 to about 0.01.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, perturb one point | empirical loss: 0.0041 -> 1.9e-5 | Helped |
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, perturb 10 points | empirical loss: 0.0045 -> 6.2e-5 | Helped |
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, stepsize 0.20 | empirical loss: 0.0045 -> 6.20 \times 10^{-5} | Helped |
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, stepsize 0.50 | empirical loss: 0.15 -> 0.005 | Helped |
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, stepsize 1 | empirical loss: 1.19 -> 0.3 | Helped |
| Use the graphical-derivative estimated solution instead of the original solution | ResNet56 · CIFAR-10, stepsize 2 | empirical loss: 3.35 -> 1.9 | Helped |
| Move the original toy-model solution to the graphical-derivative estimated solutions | Two-layer linear neural network · Toy example, poisoned data | empirical risk: 0.4 -> nearly zero (around 0.01) | Helped |
| Estimate the solution change with the graphical derivative rather than retraining | ResNet56 · CIFAR-10, 10 data perturbation scenarios | relative difference: 5.37 \times 10^{-5} (Original vs. Retrained Solution); 19 \times 10^{-5} (Original vs. Estimated Solution) | Mixed |
Limitations
- Theoretical Lipschitz-like result proven only for deep fully connected ReLU networks, not ResNets.
- Estimation accuracy degrades as perturbation stepsize grows (loss 1.9 at 2% stepsize for estimated solution).
- Assumes overparameterization and full-rank condition on the stacked Jacobians.
Experimental setup
Datasets: CIFAR-10
Benchmarks: CIFAR-10
Metrics: empirical loss on perturbed points relative difference between solutions Lipschitz modulus
Baselines: Original solution (no update) Retrained solution
Builds on: influence function Dini implicit function theorem Memory-Perturbation Equation
Closest papers by meaning
- A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness (NeurIPS 2024 · similarity 0.84)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.84)
- Set Valued Predictions For Robust Domain Generalization (ICML 2025 · similarity 0.83)
- Contrasting Adversarial Perturbations: The Space of Harmless Perturbations (AAAI 2025 · similarity 0.83)
- Expressivity of ReLU-Networks under Convex Relaxations (ICLR 2024 · similarity 0.83)
- Understanding and Diagnosing Deep Reinforcement Learning (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.