Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Deep Active Learning with Noise Stability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NoiseStability, a model-agnostic deep active learning method that scores unlabeled examples by feature deviation under small random parameter perturbations and selects diverse batches via greedy k-center.
Problem
Uncertainty estimation for deep neural networks in active learning is unreliable due to over-confidence, and existing fixes need auxiliary models or special training schemes while often ignoring batch diversity.
Approach
Measure uncertainty as the magnitude of output deviation when model parameters are randomly perturbed with small Gaussian-like noise, and use the deviation directions as projected gradients for diverse batch selection; theory shows the expected deviation norm equals the Jacobian Frobenius norm and pairwise distances/inner products are preserved in expectation.
Key results
- On image classification, NoiseStability achieves the best AUBC on Cifar10 (64.46) and SVHN (95.46) and is competitive on MNIST (83.00 vs ALFA-Mix 83.48).
- On MRPC with BERT, NoiseStability reaches AUBC 71.26 versus 70.56 for BALD and 70.30 for CoreSet.
- On Ames Housing regression, NoiseStability achieves AUBC 828.78, the lowest among methods (Random 849.45, CoreGCN 853.67).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NoiseStability instead of the strongest reported baseline | active learning · MNIST | AUBC: 83.48$\pm$0.70 -> 83.00$\pm$0.79 | Hurt |
| Use NoiseStability instead of VAAL | active learning · Cifar10 | AUBC: 63.66$\pm$1.11 -> 64.46$\pm$0.62 | Helped |
| Use NoiseStability instead of BADGE | active learning · SVHN | AUBC: 95.31$\pm$0.01 -> 95.46$\pm$0.05 | Helped |
| Use NoiseStability instead of QBC | active learning · Ames Housing | AUBC (MAE; lower is better): 840.60$\pm$18.82 -> 828.78$\pm$16.92 | Helped |
| Use NoiseStability instead of LL4AL | active learning · Cityscapes | AUBC: 53.82 -> 54.25 | Helped |
| Use NoiseStability instead of BALD | active learning · MRPC | AUBC: 70.56$\pm$3.68 -> 71.26$\pm$3.74 | Helped |
| Add noise-deviation direction to magnitude-based selection (ablation) | NoiseStability on MNIST · MNIST | classification performance | Helped |
| Add noise-deviation magnitude to direction-only selection (ablation) | NoiseStability on MNIST · MNIST | classification performance | Helped |
| Use noise direction only, normalizing each deviation to unit magnitude (ablation) | NoiseStability on MNIST · MNIST | classification performance | No clear effect |
| Use noise magnitude only, selecting maximum deviation magnitude (ablation) | NoiseStability on MNIST · MNIST | classification performance | Hurt |
| Set noise magnitude ζ too large (including 1 or 10) (ablation) | NoiseStability · MNIST | classification performance | Hurt |
| Set noise magnitude ζ too small (10^{-6}) (ablation) | NoiseStability · MNIST | classification performance | Hurt |
Surprising findings
- Random selection is very competitive after the first two cycles in the regression task, making active learning hard to beat passive learning there.
- Too small a noise magnitude is also suboptimal, since trivial perturbations activate only some neurons and make the deviation less informative.
Limitations
- Concentration and theory rely on small-noise first-order Taylor approximation and independence/zero-mean assumptions on the noise.
- Results on several tasks are averaged over only 3 runs or a single run (Cityscapes), and sensitivity to the noise magnitude zeta requires tuning.
- Gains over strong baselines are sometimes marginal, e.g. MNIST where ALFA-Mix is slightly better.
Experimental setup
Datasets: MNIST CIFAR-10 SVHN CIFAR-100 Caltech101 Ames Housing Cityscapes MRPC
Benchmarks: MNIST CIFAR-10 SVHN CIFAR-100 Caltech101 Ames Housing Cityscapes MRPC
Metrics: accuracy per AL cycle AUBC mean absolute error (regression) average ranking
Baselines: Random Entropy Least Margin BALD QBC CoreSet VAAL LL4AL BADGE CoreGCN ALFA-Mix
Builds on: BADGE Coreset linearised Laplace approximation BAIT
Closest papers by meaning
- Confidence-based Reliable Learning under Dual Noises (NeurIPS 2022 · similarity 0.88)
- SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.87)
- Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples (ICML 2024 · similarity 0.86)
- Uncertainty for Active Learning on Graphs (ICML 2024 · similarity 0.85)
- Transductive Active Learning: Theory and Applications (NeurIPS 2024 · similarity 0.85)
- Querying Easily Flip-flopped Samples for Deep Active Learning (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.