Research map / Learning theory and algorithms / Prediction and decision losses
Querying Easily Flip-flopped Samples for Deep Active Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the least disagree metric (LDM), an asymptotically consistent estimator of a sample's distance to the decision boundary via Gaussian parameter perturbation, and an LDM-seeding batch active learning algorithm (LDM-S) that combines low-LDM selection with k-means++-style diversity.
Problem
Measuring a sample's distance to the decision boundary for uncertainty-based active learning is intractable for deep multiclass classifiers, and existing approximate margin measures lack justification.
Approach
Define LDM as the infimum, over hypotheses that flip the sample's prediction, of the disagreement probability with the current model; estimate it by Gaussian perturbation of the learned parameters with multiple variances, and select batches by LDM-weighted k-means++ seeding on last-layer features.
Key results
- LDM-S attains R_LDM-S(0)=35% in the performance profile, while all other algorithms are below 15%.
- On OpenML #156, LDM-S achieves +1.18% over Random while Coreset reaches -19.53% and DBAL -14.40%.
- LDM-S runtime is roughly 3-6% above Entropy on CIFAR10, SVHN, CIFAR100, Tiny ImageNet and FOOD101.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LDM-S instead of the strongest listed baseline (ENS) | active learning · OML#6 | test accuracy difference (%): 4.43{ ±0.31} -> 4.76{ ±0.36} | Helped |
| use LDM-S instead of the strongest listed baseline (BADGE) | active learning · OML#156 | test accuracy difference (%): 1.06{ ±0.39} -> 1.18{ ±0.32} | Helped |
| use LDM-S instead of the strongest listed baseline (Margin) | active learning · OML#44135 | test accuracy difference (%): 4.11{ ±0.32} -> 4.36{ ±0.27} | Helped |
| use LDM-S instead of the strongest listed baseline (BAIT) | active learning · MNIST | test accuracy difference (%): 3.37{ ±0.43} -> 3.33{ ±0.43} | Hurt |
| use LDM-S instead of the strongest listed baseline (BADGE) | active learning · CIFAR10 | test accuracy difference (%): 0.90{ ±0.21*} -> 1.34{ ±0.19} | Helped |
| use LDM-S instead of the strongest listed baseline (DBAL) | active learning · SVHN | test accuracy difference (%): 2.46{ ±0.21} -> 2.53{ ±0.22} | Helped |
| use LDM-S instead of the strongest listed baseline (Coreset) | active learning · CIFAR100 | test accuracy difference (%): 0.89{ ±0.49} -> 0.98{ ±0.44} | Helped |
| use LDM-S instead of the strongest listed baseline (ProbCov) | active learning · Tiny ImageNet | test accuracy difference (%): 0.45{ ±0.23} -> 0.55{ ±0.16} | Helped |
| use LDM-S instead of the strongest listed baseline (Coreset) | active learning · FOOD101 | test accuracy difference (%): 1.30{ ±0.16} -> 1.27{ ±0.34} | Hurt |
| use LDM-S instead of the strongest listed baseline (BADGE) | active learning · ImageNet | test accuracy difference (%): 0.71{ ±0.81} -> 0.96{ ±0.23} | Helped |
| select by LDM instead of random sampling | active learning · 2D binary classification | test accuracy | Helped |
| select by LDM instead of entropy sampling | active learning · 2D binary classification | test accuracy | No clear effect |
Surprising findings
- Selecting purely by smallest LDM can underperform due to sampling bias; the diversity-aware seeding is needed for best results.
- Coreset underperforms random sampling substantially on several datasets (e.g., -19.53% on OpenML #156).
Limitations
- Asymptotic consistency guarantee and Gaussian-coverage assumption are proven only for 2D binary linear classifiers; deep networks rely on empirical evidence.
- Choice of the number of sampled hypotheses s is open; deep architectures may need larger s, increasing runtime.
- Gaussian sampling is an informal proxy for posterior sampling.
Experimental setup
Datasets: OpenML #6 OpenML #156 OpenML #44135 MNIST CIFAR10 SVHN CIFAR100 Tiny ImageNet FOOD101 ImageNet
Benchmarks: OpenML #6 OpenML #156 OpenML #44135 MNIST CIFAR10 SVHN CIFAR100 Tiny ImageNet FOOD101 ImageNet
Metrics: test accuracy performance profile (Dolan-More) penalty matrix runtime (min) paired t-test
Baselines: Random Entropy Margin Coreset ProbCov DBAL ENS BADGE BAIT
Builds on: k-means++ seeding disagree metric (Hanneke) EXP3 BADGE BAIT
Closest papers by meaning
- Active Learning with Neural Networks: Insights from Nonparametric Statistics (NeurIPS 2022 · similarity 0.88)
- Deep Active Learning with Noise Stability (AAAI 2024 · similarity 0.85)
- Uncertainty Herding: One Active Learning Method for All Label Budgets (ICLR 2025 · similarity 0.85)
- One-shot Active Learning Based on Lewis Weight Sampling for Multiple Deep Models (ICLR 2024 · similarity 0.84)
- Contextual Active Model Selection (NeurIPS 2024 · similarity 0.84)
- Active Classification with Few Queries under Misspecification (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.