Research map / Label-efficient and robust learning / Noisy labels
Improved Algorithm for Deep Active Learning under Imbalance via Optimal Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIRECT, a deep active learning algorithm that reduces class-imbalanced selection to per-class 1-D agnostic threshold learning (VReduce) and annotates examples around the optimal separation threshold, robust to label noise and batch labeling.
Problem
Active learning under severe class imbalance combined with label noise, where random and uncertainty sampling waste budget on majority-class examples and bisection-based methods like GALAXY are fragile to noise and cannot label in parallel.
Approach
Reduce each one-vs-rest imbalanced classification problem to a 1-D agnostic threshold-learning problem over margin-sorted examples; use a version-space procedure (VReduce, based on ACED) to find the optimal separation threshold and then annotate equal numbers of examples on each side of it.
Key results
- DIRECT reduces annotation cost by over 60% versus the best existing active learning method and over 80% versus random sampling (abstract claim).
- Under 20% label noise on imbalanced CIFAR-100 (two classes), DIRECT saves more than 60% of annotation cost versus random sampling while existing algorithms underperform random sampling.
- On imbalanced SVHN with ResNet-18, DIRECT saves more than 80% of annotation cost compared to random sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIRECT instead of the best existing active learning algorithm | Active learning on imbalanced datasets · Imbalanced datasets | annotation cost: saving more than 60% annotation cost | Helped |
| Use DIRECT instead of random sampling | Active learning on imbalanced SVHN · Imbalanced SVHN | annotation cost: save more than 80% of the annotation cost | Helped |
| Use DIRECT instead of baseline algorithms | Active learning under imbalance and label noise · Imbalanced datasets with 10% label noise | balanced accuracy | Helped |
| Use DIRECT instead of GALAXY | Active learning on imbalanced CIFAR-100 · Imbalanced CIFAR-100, two classes, no label noise | generalization accuracy | Hurt |
| Use DIRECT instead of random sampling | Active learning on imbalanced CIFAR-100 · Imbalanced CIFAR-100, two classes, 20% label noise | annotation cost: saving more than 60% of the annotation cost | Helped |
| Use DIRECT with parallel annotation batch size 5 instead of 1 | DIRECT · Imbalanced datasets | balanced accuracy | No clear effect |
| Use DIRECT instead of baseline algorithms on less frequent classes | Active learning on ImageNet-LT · ImageNet-LT, class indices 301-1000 | balanced accuracy | Helped |
| Use DIRECT instead of baseline algorithms on more frequent classes | Active learning on ImageNet-LT · ImageNet-LT, more frequent classes | accuracy | Hurt |
Surprising findings
- Existing imbalance-aware active learning methods such as GALAXY and SIMILAR can underperform random sampling under high label noise (20%).
- Annotating many minority-class examples (e.g., Most Likely Positive, SIMILAR) does not translate to better balanced accuracy, showing class-balancedness must be traded off against informativeness.
Limitations
- Asynchronous labeling is not addressed and is left as future work.
- Experiments use synthetic uniform random label noise only; other noise models are not studied.
- DIRECT slightly underperforms GALAXY in generalization accuracy in the noiseless two-class CIFAR-100 setting.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN PathMNIST FMoW iWildCam ImageNet-LT iNaturalist CIFAR-10LT CIFAR-100LT
Benchmarks: Imbalanced CIFAR-10 Imbalanced CIFAR-100 Imbalanced SVHN PathMNIST FMoW (LabelBench) iWildCam (LabelBench) ImageNet-LT iNaturalist
Metrics: balanced accuracy balanced pool accuracy annotation cost savings
Baselines: GALAXY SIMILAR BADGE BASE BAIT Cluster Margin Confidence Sampling Most Likely Positive Random Sampling Margin Sampling CoreSet
Builds on: GALAXY ACED LabelBench BADGE
Closest papers by meaning
- Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance (ICML 2026 · similarity 0.91)
- Rethinking the Value of Labels for Improving Class-Imbalanced Learning (NeurIPS 2020 · similarity 0.89)
- Instance-wise Supervision-level Optimization in Active Learning (CVPR 2025 · similarity 0.87)
- Imbalances in Neurosymbolic Learning: Characterization and Mitigating Strategies (NeurIPS 2025 · similarity 0.85)
- Learning Imbalanced Data with Beneficial Label Noise (ICML 2025 · similarity 0.85)
- Inconsistency-Based Data-Centric Active Open-Set Annotation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.