Research map / Label-efficient and robust learning / Recognition and long-tail
A Cross-Domain Benchmark for Active Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CDALBench, a cross-domain active learning benchmark (image, text, tabular, plus synthetic and pre-encoded data) with a greedy oracle and 50 repetitions per experiment, showing rankings vary across domains and that few runs give unreliable conclusions.
Problem
Active learning evaluations are narrow (single domain) and use too few repetitions, so reported lifts over baselines generalize poorly and can be misleading due to high run-to-run variance.
Approach
A unified AL benchmark spanning image, text and tabular data with pre-encoded semi-supervised variants, a cheap greedy oracle (sampling tau=20 candidates per step, falling back to margin sampling), a reproducible seeding scheme, and 50 runs per experiment, plus two synthetic datasets (Honeypot, Diverging Sine) targeting uncertainty and clustering shortcomings.
Key results
- Margin sampling is the top non-oracle method on tabular and text domains, while on images least confident sampling is strong; the top-3 non-oracle methods differ for any two domains.
- The greedy oracle outperforms all AL methods by about 5% on average over the best method per dataset (tau=20).
- With only 3 runs sampled from 50, per-dataset rank differences of up to 4.7 positions appear (e.g., TypiClust on Splice), and BALD and Entropy swap 2 overall places.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use margin sampling rather than random sampling | Active learning on real-world datasets · Splice, DNA, USPS, Cifar10, FMnist, TopV2, News | average rank | Helped |
| use least-confident sampling rather than margin sampling | Active learning on image data · Cifar10 and FMnist | average rank | Mixed |
| use margin sampling rather than least-confident sampling | Active learning on text and tabular data · Text and tabular domains | performance | Mixed |
| use uncertainty sampling rather than clustering-based Coreset or TypiClust | Active learning on Diverging Sine · Diverging Sine | rank | Helped |
| use BADGE rather than clustering-based Coreset or TypiClust | Active learning on Diverging Sine · Diverging Sine | rank | Helped |
| cluster raw data instead of classifier embeddings (ablation) | Coreset and TypiClust · Diverging Sine | rank | Hurt |
| evaluate with only 3 repetitions rather than many repetitions | Active learning experiments · Splice and real-world datasets | average rank | Mixed |
| use the greedy oracle instead of current AL methods | Active learning · Each dataset | AUC performance: +5% | Helped |
Surprising findings
- Image domain behaves differently from text and tabular: least confident sampling is best on images but not in any other domain.
- With 3 runs, a well-established method can be significantly better or worse than random on the same dataset depending on the seed.
Limitations
- Smaller classifiers (MLP, linear, BiLSTM, ResNet18) and fixed validation splits sampled from full dataset make results not directly transferable to practical settings.
- Greedy oracle may underestimate the true upper bound; Cifar10 and FashionMNIST oracle curves are forecast beyond 2000 iterations.
- Text datasets only offered in encoded form; no data augmentation and no class-imbalance study.
Experimental setup
Datasets: Splice DNA USPS FashionMNIST Cifar10 TopV2 News Category Honeypot Diverging Sine
Benchmarks: CDALBench (Splice, DNA, USPS, FashionMNIST, Cifar10, TopV2, News, Honeypot, Diverging Sine)
Metrics: normalized AUC of accuracy curve average rank critical difference diagram (Wilcoxon signed-rank)
Baselines: Random Margin Entropy Least confident sampling BALD BADGE Coreset TypiClust CoreGCN DSA LSA Galaxy
Builds on: SimCLR Ji et al. recommendations Lueth et al. pitfalls margin sampling BADGE
Closest papers by meaning
- Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees (AAAI 2024 · similarity 0.89)
- Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment (NeurIPS 2023 · similarity 0.88)
- Active Finetuning: Exploiting Annotation Budget in the Pretraining-Finetuning Paradigm (CVPR 2023 · similarity 0.87)
- Towards Cost-Effective Learning: A Synergy of Semi-Supervised and Active Learning (CVPR 2025 · similarity 0.86)
- CoLAL: Co-learning Active Learning for Text Classification (AAAI 2024 · similarity 0.86)
- Instance-wise Supervision-level Optimization in Active Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.