Research map / Learning theory and algorithms / Prediction and decision losses
Active Learning Guided by Efficient Surrogate Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a batch-mode active learning method that uses an incrementally updated sparse Gaussian process surrogate to emulate continuous retraining of a ResNet101 learner, combining an influence utility and an uncertainty utility.
Problem
Batch-based active learning with deep networks causes redundant sampling because retraining the network after every single label is computationally impractical.
Approach
A Gaussian process surrogate with a product kernel over inputs and learner outputs is updated after every new label, providing an influence utility (expected reduction in predictive entropy over the unlabeled set, computed via Sherman-Morrison-Woodbury) and an uncertainty utility (calibrated softmax entropy propagated between DNN retraining stages), combined by a convex weight based on estimated t
Key results
- On CIFAR100, Caltech256 and Tiny-ImageNet the method yields significant accuracy gains over baselines, while on FashionMNIST it is on par with the best methods (WS and BADGE).
- Per-iteration selection cost is about 0.35 seconds on FashionMNIST, comparable to BADGE (0.58 s) and far below VAAL and TA-VAAL (about 23.7 s each).
- Using the combined product kernel instead of input-only kernel reduced mean absolute deviation between surrogate and DNN by 34% on FashionMNIST.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the product kernel combining input and learner-output kernels instead of the input kernel alone (ablation) | GP surrogate · FashionMNIST training set, 4,500 labels | mean absolute deviation: 34% | Helped |
| Use u² instead of u³ (ablation) | Active learning utility · FashionMNIST | final classification accuracy: 0.16% lower | Hurt |
| Use u² instead of u³ (ablation) | Active learning utility · FashionMNIST | label selection speed: approximately 50 times faster | Helped |
| Increase the number of basis points K to 2,000 (ablation) | Ours · CIFAR10 | accuracy: up to approximately 0.06% | Helped |
| Vary the input-kernel width by a factor in [0.3,8] (ablation) | Ours · CIFAR10 | accuracy | No clear effect |
| Decrease the input-kernel width magnification factor below 0.1 (ablation) | Ours · CIFAR10 | accuracy | Hurt |
| Vary the noise level σ² within {10⁻⁸,10⁻⁹,10⁻¹¹,10⁻¹²} (ablation) | Ours · CIFAR10 | accuracy | No clear effect |
| Replace the accuracy-weighted utility combination with a uniform combination of u¹ and u² (ablation) | Ours · CIFAR10 | accuracy | Hurt |
| Use u¹-only instead of combining u¹ and u² (ablation) | Ours · CIFAR100 and early-stage FashionMNIST | accuracy | Mixed |
| Use u²-only instead of combining u¹ and u² (ablation) | Ours · Later-stage FashionMNIST | accuracy | Hurt |
| Combine influence and uncertainty utilities with the proposed accuracy-weighted combination rather than using either utility alone (ablation) | Ours · CIFAR100, Caltech256, Tiny-ImageNet, FashionMNIST | accuracy | Helped |
Surprising findings
- The uncertainty utility u2 outperformed the more principled influence-based predictive entropy reduction u3 in practice, with u2 only 0.16% lower in accuracy while about 50 times faster.
- Predictive-entropy-based selection tends to pick outliers, which the proposed uncertainty utility avoids because ReLU/sigmoid networks are overconfident on outliers.
Limitations
- The surrogate GP may deviate from the DNN between retraining stages, and no theoretical analysis of surrogate quality is provided.
- Hyperparameters such as K, sigma_x, and sigma_f are fixed heuristically without tuning on validation labels, and the test-accuracy estimate used for combining utilities is approximate.
- Influence utility u1 is agnostic to the task labels, so it relies on the uncertainty utility to adapt to later learning stages.
Experimental setup
Datasets: Tiny-ImageNet CIFAR100 FashionMNIST Caltech256 CIFAR10
Benchmarks: Tiny-ImageNet CIFAR100 FashionMNIST Caltech256
Metrics: classification accuracy accuracy difference relative to final algorithm
Baselines: CoreSet LearningLoss BADGE SeqGCN GradNorm WS VAAL TA-VAAL Random
Builds on: BADGE BALD CoreSet LearningLoss
Closest papers by meaning
- Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation (NeurIPS 2022 · similarity 0.89)
- Towards Robust and Reproducible Active Learning Using Neural Networks (CVPR 2022 · similarity 0.88)
- Online Active Learning with Surrogate Loss Functions (NeurIPS 2021 · similarity 0.87)
- One-shot Active Learning Based on Lewis Weight Sampling for Multiple Deep Models (ICLR 2024 · similarity 0.85)
- Active Learning for Continual Learning: Keeping the Past Alive in the Present (ICLR 2025 · similarity 0.85)
- Instance-wise Supervision-level Optimization in Active Learning (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.