Research map / Learning theory and algorithms / Combinatorial and search optimization
AutoAL: Automated Active Learning with Differentiable Query Strategy Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoAL, a differentiable bi-level framework (SearchNet and FitNet) that searches over and combines existing active learning query strategies to pick the best strategy per task for deep active learning.
Problem
Choosing which active learning query strategy works best for a given dataset and budget is hard, and prior strategy-selection methods are computationally costly and non-differentiable.
Approach
Relaxes the discrete choice among candidate AL strategies into a continuous, sigmoid-weighted mixture and trains SearchNet and FitNet jointly in a differentiable bi-level optimization using only labeled data, so that SearchNet scores unlabeled samples by the strategies that agree on them.
Key results
- AutoAL is reported to consistently outperform all seven candidate AL strategies and the deep AL baselines across the seven datasets in accuracy-vs-labels curves.
- AutoAL shows smaller standard deviations across three repeated runs than baselines, per the authors.
- With three candidate strategies, AutoAL's total runtime is reported as about 1.3x that of EntropySampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoAL instead of the individual baseline AL strategies | Active learning with ResNet-18 · CIFAR-10, CIFAR-100, SVHN, TinyImageNet, OrganCMNIST, PathMNIST, and TissueMNIST | accuracy | Helped |
| Remove the ResNet-18 backbone and optimize solely with the loss prediction module (ablation) | AutoAL SearchNet · CIFAR-100, SVHN, and OrganCMNIST | accuracy | Hurt |
| Add a loss prediction module to the ResNet-18 backbone (ablation) | AutoAL SearchNet · CIFAR-100, SVHN, and OrganCMNIST | accuracy | Helped |
| Expand the candidate pool from one strategy to multiple strategies (ablation) | AutoAL · CIFAR-100 and OrganCMNIST | accuracy | Helped |
| Increase the number of AL candidates in the selection pool (ablation) | AutoAL · SVHN | accuracy | Helped |
| Increase the number of AL candidates in the selection pool (ablation) | AutoAL · CIFAR-100 and OrganCMNIST | accuracy | No clear effect |
| Use three candidate AL strategies rather than a single candidate (ablation) | AutoAL · CIFAR-100, SVHN, and OrganCMNIST | accuracy | Helped |
| Use three candidates in the AL selection pool (ablation) | AutoAL · Active learning across the evaluated datasets | runtime (relative to EntropySampling): 1.3x | Hurt |
Surprising findings
- Using only the loss prediction module to optimize the network (without the ResNet backbone update) causes a significant performance drop.
- KMeans sampling dominates early AL rounds and is de-prioritized later, while uncertainty measures gain weight in final rounds on CIFAR-100.
Limitations
- Total runtime depends heavily on the cost of the candidate AL strategies, with some candidates (e.g., ENSvarR) much slower.
- Performance upper bound on the candidate pool size varies by dataset, so the number of candidates requires tuning.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 SVHN TinyImageNet OrganCMNIST PathMNIST TissueMNIST
Benchmarks: CIFAR-10 CIFAR-100 SVHN TinyImageNet OrganCMNIST PathMNIST TissueMNIST
Metrics: test accuracy vs. labeled data standard deviation over three runs runtime relative to EntropySampling
Baselines: Maximum Entropy Margin Sampling Least Confidence KMeans Sampling BALD VarRatio MeanSTD BADGE LPL VAAL Coreset ENSvarR
Builds on: loss prediction active learning BALD BADGE Gaussian mixture model
Closest papers by meaning
- Task-Aware Variational Adversarial Active Learning (CVPR 2021 · similarity 0.86)
- Towards Robust and Reproducible Active Learning Using Neural Networks (CVPR 2022 · similarity 0.86)
- One-shot Active Learning Based on Lewis Weight Sampling for Multiple Deep Models (ICLR 2024 · similarity 0.85)
- Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples (ICML 2024 · similarity 0.84)
- A Cross-Domain Benchmark for Active Learning (NeurIPS 2024 · similarity 0.84)
- Entropic Open-Set Active Learning (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.