Research map / Large language models / Language, safety and interpretability
Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POCA and its Bayesian instantiation µPOCA, which extends BALD/EPIG to partially observed tabular data by imputing unobserved features with LLM-based generative surrogate models, for cost-aware label and feature acquisition.
Problem
Active data acquisition when training data is partially observed in features and acquisition of features and labels is costly; standard active learning assumes fully observed pool and training sets.
Approach
Extends expected information gain (BALD) and expected predictive information gain (EPIG) to partially observed settings (PO-EIG, PO-EPIG) by computing expectations over features imputed by an LLM generative surrogate model, with theory showing joint label-and-feature acquisition yields at least as much uncertainty reduction as label-only acquisition.
Key results
- PO-EIG and PO-EPIG generally match or outperform their fully observed counterparts (BALD, EPIG) across all five tabular datasets, with plain EIG on partially observed features sometimes worse than random on Magic.
- In cost-aware experiments on Magic, PO-EIG with 20% feature acquisition is best when feature cost dominates, while 60% feature acquisition is best when label cost is 50% of total.
- Theory (Corollary 1) shows the expected uncertainty reduction from acquiring labels plus features is at least that of acquiring labels alone, with equality under conditional independence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PO-EIG instead of vanilla EIG with partially observed features | Active learning acquisition on Magic, Adult, Banking, Cardio, and Housing · Magic, Adult, Banking, Cardio, Housing | test accuracy | Helped |
| Use PO-EPIG instead of vanilla EPIG with partially observed features | Active learning acquisition on Magic, Adult, Banking, Cardio, and Housing · Magic, Adult, Banking, Cardio, Housing | test accuracy | Helped |
| Acquire features rather than relying only on imputation (ablation) | µPOCA on Magic · Magic | accuracy (%): 74.3 -> 83.1 | Helped |
| Acquire features rather than relying only on imputation (ablation) | µPOCA with LLM imputation · Magic | accuracy (%): 76.6 -> 83.1 | Helped |
| Use LLM imputation instead of zero-value imputation when acquiring 0 features (ablation) | µPOCA on Magic · Magic | accuracy (%): 74.3 -> 76.6 | Helped |
| Use LLM imputation instead of zero-value imputation while acquiring features (ablation) | µPOCA on Magic · Magic | accuracy | Mixed |
| Acquire only a subset of features rather than all features (ablation) | PO-EIG on Magic under an acquisition budget · Magic | test accuracy | Mixed |
| Use selective feature acquisition under a budget instead of vanilla EIG | PO-EIG on Magic · Magic | test accuracy | Helped |
Surprising findings
- Partially observed EIG computed with imputed features can be significantly worse than random selection on the Magic dataset.
- Imputation alone cannot fully replace direct feature acquisition; acquiring more features improves performance even with imputation available.
Limitations
- Does not assess how acquisition restrictions are selected.
- LLM-based imputation may carry biases that affect acquisition; not studied.
- Only a 7B-parameter LLM was used; larger models not explored.
Experimental setup
Datasets: Magic Adult Housing Cardio Banking
Benchmarks: Magic Adult Housing Cardio Banking
Metrics: test accuracy 95% confidence interval
Baselines: BALD (EIG) EPIG Random Oracle (fully observed acquisition) Zero value imputation
Builds on: BALD EPIG Active Feature Acquisition with generative surrogate models Bayesian experimental design
Closest papers by meaning
- Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.88)
- Stochastic Encodings for Active Feature Acquisition (ICML 2025 · similarity 0.87)
- Towards Cost-Effective Learning: A Synergy of Semi-Supervised and Active Learning (CVPR 2025 · similarity 0.87)
- Contextual Active Model Selection (NeurIPS 2024 · similarity 0.85)
- Information Templates: A New Paradigm for Intelligent Active Feature Acquisition (NeurIPS 2026 · similarity 0.85)
- Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.