Research map / Probabilistic and scientific modeling / Variational and generative sampling
Debiased Active Learning with Variational Gradient Rectifier
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VaGeRy, an asynchronous active-learning rectifier that quantifies sampling bias via variational latent representations and corrects it with uncertainty consistency regularization and fluctuation restriction built from training dynamics.
Problem
Active learning's preference for hard samples introduces dynamically evolving sampling bias that destabilizes training and slows convergence; existing methods neither quantify nor directly rectify this bias.
Approach
A rectifier module (memory, bias variable from variational latent gaps, receptance) trained on unlabeled data with a delayed, alternating schedule against an EMA historical teacher; it rectifies predictions used for uncertainty-based selection, with losses on uncertainty consistency and prediction fluctuation.
Key results
- VaGeRy improves accuracy in almost all 36 text configurations (six uncertainty methods across six datasets), with about 80% of checkpoints improved.
- On NoiseStability, VaGeRy improves 9 of 10 checkpoints on SST-5 and 7 of 10 on TREC.
- Entropy-based VaGeRy on CIFAR-10/100 outperforms CoreSet, Core-GCN and Learning Loss and is comparable to TiDAL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VaGeRy rectification to MS | MS active learning · SST-2, cycle 10 | accuracy: 87.56 -> 87.83 | Helped |
| Apply VaGeRy rectification to Entropy | Entropy active learning · AGNews, cycle 10 | accuracy: 88.76 -> 88.65 | Hurt |
| Apply VaGeRy rectification to RoC | RoC active learning · SST-5, cycle 10 | accuracy: 48.23 -> 48.16 | Hurt |
| Apply VaGeRy rectification to MS | MS active learning · PubMed, cycle 10 | accuracy: 80.36 -> 80.54 | Helped |
| Apply VaGeRy rectification to LC | LC active learning · TREC, cycle 10 | accuracy: 98.24 -> 98.70 | Helped |
| Apply VaGeRy rectification to RoC | RoC active learning · Newsgroups, cycle 10 | accuracy: 62.98 -> 65.34 | Helped |
Surprising findings
- Gains from VaGeRy depend on baseline strength; some configurations show drops (e.g., BALD on AGNews at 2-th cycle, -0.85).
- On the NLI OOD dataset, intermediate rounds show suboptimal performance with VaGeRy, attributed to an inadequately controlled ratio of unlabeled data.
Limitations
- Improvements are inconsistent across configurations; some checkpoints degrade.
- OOD NLI results are suboptimal in intermediate rounds, possibly due to unlabeled data ratio control.
- Diversity-aware baselines are argued to be crude, a claim not deeply tested.
Experimental setup
Datasets: SST-2 AGNews PubMed TREC SST-5 Newsgroups CIFAR-10 CIFAR-100 SA NLI ANLI Twitter
Benchmarks: SST-2 AGNews PubMed TREC SST-5 Newsgroups CIFAR-10 CIFAR-100 SA NLI ANLI
Metrics: accuracy relative improvement
Baselines: Entropy LC MoC MS RoC BALD Core-set Core-GCN Learning Loss NoiseStability TiDAL CounterAL
Builds on: Variational Adversarial Active Learning Mean Teacher Dataset Cartography CounterAL Core-set
Closest papers by meaning
- On Statistical Bias In Active Learning: How and When to Fix It (ICLR 2021 · similarity 0.87)
- Characterizing the Edge of Stability in Variational Training Without Priors (NeurIPS 2026 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.84)
- Transductive Active Learning: Theory and Applications (NeurIPS 2024 · similarity 0.84)
- Deep Active Learning with Noise Stability (AAAI 2024 · similarity 0.84)
- Efficient Backpropagation with Variance Controlled Adaptive Sampling (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.