Research map / Label-efficient and robust learning / Recognition and long-tail
Contextual Active Model Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAMS, an online contextual active model selection algorithm that combines exponential-weight policy selection over context-dependent policies with an uncertainty-based adaptive label query rule, with regret and query-complexity bounds for stochastic and adversarial streams.
Problem
Selecting among pre-trained classifiers online for streaming unlabeled contexts when labels are costly and must be requested, with both stochastic and adversarial data streams.
Approach
Combines contextual model-selection policies (plus constant single-model policies) via exponential weights, and queries labels with probability set by a model-disagreement function over the weighted prediction distribution, using importance-weighted loss estimates so that regret and query complexity are bounded in both stochastic and adversarial settings.
Key results
- CAMS requires less than 10% of the labeling effort compared with existing methods on CIFAR10 and DRIFT while achieving similar or better accuracy (abstract claim).
- On VERTEBRAL, CAMS outperforms the best baseline in query cost by about 20% despite 11 of 17 policies giving malicious or random advice.
- On CIFAR10, active query reduces query cost by 71% vs. variance-based (Model Picker) and 95% vs. random querying.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAMS contextual model selection and adaptive querying instead of the compared baselines | CAMS · CIFAR10, DRIFT, VERTEBRAL, HIV | cumulative loss versus query cost | Helped |
| Use CAMS rather than existing methods | CAMS · CIFAR10 and DRIFT | labeling effort: less than 10% | Helped |
| Use CAMS instead of the best baseline | CAMS · VERTEBRAL | query cost: 20% | Helped |
| Use CAMS instead of the variance-based query strategy (ablation) | CAMS recommendation strategy with alternative query strategies · CIFAR10 | query cost: 71% | Helped |
| Use CAMS instead of random querying (ablation) | CAMS recommendation strategy with alternative query strategies · CIFAR10 | query cost: 95% | Helped |
| Replace CAMS adaptive querying with variance-based querying (ablation) | CAMS · CIFAR10 | cumulative loss convergence rate | Hurt |
| Replace CAMS adaptive querying with random querying (ablation) | CAMS · CIFAR10 | cumulative loss convergence rate | Hurt |
| Remove context and experts (ablation) | CAMS · Context-free environment | best-classifier identification performance | No clear effect |
| Give algorithms the stream length and optimize query probabilities to allocate a fixed budget to the most informative labels (ablation) | CAMS and baselines · VERTEBRAL and HIV | cumulative loss versus query cost | Helped |
| Use CAMS weighted-majority model recommendation rather than selecting the best single policy | CAMS · HIV | cumulative loss | Helped |
Surprising findings
- CAMS outperforms the best single policy (oracle) on VERTEBRAL and HIV with fewer than 10 and 20 queries respectively.
Limitations
- Analysis restricted to classification with 0-1 loss; does not readily extend to non-uniform losses or other tasks.
- Regret bound can degrade to T^{3/4} when the data-dependent term min-gap is zero; practice relies on policy regularization.
- Theoretical bounds depend on assumptions such as a unique best policy and positive minimal selection probability.
Experimental setup
Datasets: CIFAR10 DRIFT VERTEBRAL HIV CovType
Benchmarks: CIFAR10 DRIFT VERTEBRAL HIV CovType
Metrics: cumulative loss query cost regret query complexity
Baselines: Oracle CQBC CIWAL Random Query Strategy Model Picker Query by Committee IWAL
Builds on: Model Picker EXP4 Hedge query by committee importance weighted active learning
Closest papers by meaning
- Active Testing: Sample-Efficient Model Evaluation (ICML 2021 · similarity 0.87)
- Instance-wise Supervision-level Optimization in Active Learning (CVPR 2025 · similarity 0.86)
- Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios (NeurIPS 2024 · similarity 0.85)
- Towards Cost-Effective Learning: A Synergy of Semi-Supervised and Active Learning (CVPR 2025 · similarity 0.85)
- Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.85)
- Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.