Research map / Label-efficient and robust learning / Recognition and long-tail
Harnessing the Power of Beta Scoring in Deep Active Learning for Multi-Label Text Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BESRA, a deep active learning acquisition strategy for multi-label text classification that uses the Beta family of proper scoring rules inside the Expected Loss Reduction/BEMPS framework and k-Means diverse batch selection.
Problem
Active learning for multi-label text classification under expansive label spaces and imbalanced label distributions, where existing acquisition functions (MMC, ADAPTIVE, AUDI, CVIRS, SHLR, CSRPE, GPB2M) do not explicitly handle label imbalance.
Approach
Generalizes BEMPS (expected proper score change) by replacing Brier/logarithmic scores with the asymmetric Beta family of proper scoring rules applied per label via Binary Relevance; query vectors of expected score changes are clustered with k-Means to select diverse batches. Setting alpha=0.1, beta=3 penalizes false negatives more, which helps under imbalance.
Key results
- BESRA with alpha=0.1, beta=3 is reported to consistently outperform the other AL baselines across six real MLTC datasets with TextCNN, TextRNN and BERT backbones.
- On synthetic RCV1-derived datasets, the Beta score (alpha=0.1, beta=3) shows the largest gains over Brier and logarithmic scores at high imbalance (RCV1-200, RCV1-400).
- ADAPTIVE is the strongest baseline overall, while MMC is the weakest across most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BESRA with Beta scoring (α=0.1, β=3) instead of the baseline acquisition methods | BERT active learning · Bibtex, TMC2007, Delicious, RCV1, Eurlex, Yahoo | Micro F1 | Helped |
| Use BESRA instead of other acquisition methods | TextCNN and TextRNN active learning · RCV1 | Micro F1 | Helped |
| Use Beta scoring with α=0.1, β=3 instead of Brier or Logarithmic scoring (ablation) | BERT active learning · Synthetic RCV1-10, RCV1-50, RCV1-200, RCV1-400 | Micro F1 | Helped |
| Replace Beta scoring (α=0.1, β=3) with Brier scoring (α=1, β=1) (ablation) | BESRA · Synthetic datasets | Micro F1 | Hurt |
| Replace Beta scoring (α=0.1, β=3) with Logarithmic scoring (α=0, β=0) (ablation) | BESRA · Synthetic datasets | Micro F1 | Hurt |
| Use Brier scoring (α=1, β=1) instead of Logarithmic scoring (α=0, β=0) (ablation) | BESRA · Synthetic RCV1-50 | Micro F1 | Helped |
| Use Brier rather than Logarithmic scoring (ablation) | BESRA · Other synthetic scenarios | Micro F1 | No clear effect |
| Set α=1, β=0.1, which emphasizes false-positive penalization (ablation) | BESRA · Bibtex, RCV1, Yahoo | Micro F1 | Hurt |
| Set α=0.1, β=1 for light false-negative penalization (ablation) | BESRA · Bibtex, RCV1, Yahoo | Micro F1 | No clear effect |
| Set α=0.1, β=10 for stringent false-negative penalization (ablation) | BESRA · Highly imbalanced real datasets | Micro F1 | Helped |
| Set α=0.1, β=10 rather than the moderate setting α=0.1, β=3 (ablation) | BESRA · Real-world datasets | Micro F1 | Hurt |
| Use a smaller acquisition batch (50 rather than 100) (ablation) | BESRA · RCV1, early acquisition stages | Micro F1 | Helped |
Surprising findings
- Brier and logarithmic scores, which are the standard BEMPS choices, consistently underperform the asymmetric Beta scores across imbalance levels.
- The more stringent setting beta=10 improves over moderate settings only partially and does not necessarily give the best learner, so moderate penalization (beta=3) is best.
Limitations
- Optimal alpha and beta must be selected per problem; the paper finds no universal setting and notes combinations are not explored per dataset.
- Acquisition relies on an ensemble of five models and an estimation pool, adding computational overhead.
- Results for generalization beyond the main datasets are mostly deferred to the appendix.
Experimental setup
Datasets: RCV1 Eurlex Bibtex Delicious Yahoo (health) TMC2007 Synthetic RCV1-derived MeanIR datasets (10, 50, 200, 400)
Benchmarks: Eurlex RCV1 Bibtex Delicious Yahoo (health) TMC2007 synthetic RCV1 MeanIR datasets
Metrics: micro-F1 macro-F1 precision recall precision@5 recall@5
Baselines: Random MMC ADAPTIVE AUDI CVIRS SHLR CSRPE GPB2M
Builds on: BEMPS Expected Loss Reduction MMC Binary Relevance
Closest papers by meaning
- ADIOS: Architectures Deep In Output Space (ICML 2016 · similarity 0.86)
- AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification (NeurIPS 2019 · similarity 0.86)
- Compositional Generalization for Multi-Label Text Classification: A Data-Augmentation Approach (AAAI 2024 · similarity 0.85)
- Active Discriminative Text Representation Learning (AAAI 2017 · similarity 0.85)
- Distributional Semantics Meets Multi-Label Learning (AAAI 2019 · similarity 0.85)
- CoLAL: Co-learning Active Learning for Text Classification (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.