Research map / Optimization / Optimizers
AdaptGrad: Adaptive Sampling to Reduce Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaptGrad, an adaptive SmoothGrad variant that sets per-pixel Gaussian noise variance so that a chosen fraction of samples stays within the data range, reducing out-of-range sampling noise in gradient saliency maps.
Problem
SmoothGrad's noise variance sigma is set manually or heuristically, causing many Gaussian samples to fall outside the valid input range [x_min, x_max], which introduces extra noise into smoothed gradient saliency maps.
Approach
Reinterprets SmoothGrad as a convolution over a bounded domain and derives a per-dimension sigma from the distance to the nearest data bound and an extra-noise level c (via inverse erf), so out-of-range sampling is controlled while keeping the smoothing convergent.
Key results
- AdaptGrad improves Sparseness over SmoothGrad for all three models and all five explanation variants; e.g., on VGG16 SG 0.5289 vs AG 0.5740 (Grad-based).
- On VGG16 with selective search, localization precision rises from 0.01937 (SmoothGrad) to 0.05951 (AdaptGrad).
- AdaptGrad adds negligible runtime over SmoothGrad (VGG16: 0.6054 s vs 0.5914 s per 1,000 images as reported).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, VGG16 | Sparseness: 0.5289 -> 0.5740 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, InceptionV3 | Sparseness: 0.5369 -> 0.5584 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, ResNet50 | Sparseness: 0.5614 -> 0.5721 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, VGG16 | Faithfulness-I: 0.6729 -> 0.6748 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, InceptionV3 | Faithfulness-I: 0.5955 -> 0.6145 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · ImageNet, ResNet50 | Faithfulness-I: 0.2626 -> 0.2692 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · Object localization, ImageNet, VGG16 | Localization precision: 0.01937 -> 0.05951 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · Object localization, ImageNet, InceptionV3 | Localization precision: 0.1048 -> 0.1237 | Helped |
| Use adaptive sampling with extra-noise level c=0.95 instead of SmoothGrad sampling | Gradient smoothing · Object localization, ImageNet, ResNet50 | Localization precision: 0.05528 -> 0.08108 | Helped |
| Clip sampled values to [x_min, x_max] rather than using adaptive sampling (ablation) | AdaptGrad · ImageNet, VGG16 | Sparseness: 0.5740(0.0000) vs 0.5294(0.0001) | Hurt |
| Clip sampled values to [x_min, x_max] rather than using adaptive sampling (ablation) | AdaptGrad · ImageNet, InceptionV3 | Sparseness: 0.5584(0.0001) vs 0.5441(0.0001) | Hurt |
| Clip sampled values to [x_min, x_max] rather than using adaptive sampling (ablation) | AdaptGrad · ImageNet, ResNet50 | Sparseness: 0.5721(0.0001) vs 0.5607(0.0001) | Hurt |
Surprising findings
- Faithfulness insertion/deletion metrics are argued to be unreliable: noisier saliency maps can achieve deceptively higher deletion scores.
- Hard clipping of samples to the data range (ClipGrad) performs worse than AdaptGrad on Sparseness and consistency, and the paper argues hard thresholds hinder smoothing convergence.
Limitations
- The extra-noise level c remains an empirical choice without task-specific guidance.
- Noise is measured only indirectly via Sparseness; no direct noise metric is established.
- Faithfulness metrics are argued to be unreliable, limiting the evidence for the noise-out-of-range relationship.
Experimental setup
Datasets: ILSVRC2012 (ImageNet) MNIST
Benchmarks: ILSVRC2012 (ImageNet) MNIST
Metrics: Sparseness (Gini index) Faithfulness insertion/deletion AUC Consistency (data randomization sanity check) Invariance SIC object localization precision (selective search) adversarial pixel count (FGSM) execution time
Baselines: SmoothGrad Vanilla Gradient NoiseGrad ClipGrad Gradient x Input Integrated Gradients
Builds on: SmoothGrad NoiseGrad Integrated Gradients Gradient x Input
Closest papers by meaning
- Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models (ICML 2024 · similarity 0.85)
- Interpreting and Improving Diffusion Models from an Optimization Perspective (ICML 2024 · similarity 0.85)
- EVODiff: Entropy-aware Variance Optimized Diffusion Inference (NeurIPS 2025 · similarity 0.85)
- Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models (ICLR 2025 · similarity 0.84)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.84)
- An Edit Friendly DDPM Noise Space: Inversion and Manipulations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.