Research map / Large language models / Language, safety and interpretability
IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDEAL, an influence-driven selective annotation method that builds a kNN directed graph over unlabeled data, scores subsets by independent-cascade diffusion, and greedily selects annotation examples for in-context learning.
Problem
Reducing the annotation cost of in-context learning by selecting which unlabeled examples to annotate, so that prompts retrieved from the annotated subset perform well.
Approach
Quantify the coverage of a candidate unlabeled subset as the expected number of nodes activated by an independent-cascade diffusion on a kNN similarity graph, and greedily maximize this marginal influence, avoiding an explicit diversity-representativeness trade-off and any LLM inference during selection.
Key results
- With annotation budget 18, IDEAL beats Vote-k on 17 of 18 dataset/budget cases (e.g., MRPC 63.0 vs 61.1, Xsum 19.6 vs 15.2 for GPT-J).
- IDEAL achieves on average a 7.8x speedup in subset selection over Vote-k, since it needs no LLM predictions on unlabeled data.
- Theory: greedy selection achieves f(S_m) >= (1-(1-1/m)^m) f(S*_m), approaching 1-1/e.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IDEAL instead of Vote-k for selective annotation | GPT-J 6B and Text-devinci-002 ICL · Main benchmarks, budget 18 | accuracy / task score: 17 out of 18 cases | Helped |
| Use IDEAL instead of Vote-k for selective annotation | GPT-J 6B and Text-devinci-002 ICL · Main benchmarks, budget 100 | accuracy / task score | Mixed |
| Use IDEAL instead of Vote-k for subset selection | Selective annotation · All tasks (annotation budget 18) | selection time: 7.8× lead on average | Helped |
| Use similarity-based rather than random prompt retrieval (ablation) | IDEAL · MRPC | accuracy: 62.5 -> 66.4 | Helped |
| Use similarity-based rather than random prompt retrieval (ablation) | IDEAL · MNLI | accuracy: 39.0 -> 41.0 | Helped |
| Use similarity-based rather than random prompt retrieval (ablation) | IDEAL · HellaSwag | accuracy: 66.8 -> 68.6 | Helped |
| Expand IDEAL candidate prompts with automatically annotated examples (Auto-IDEAL) (ablation) | IDEAL · MNLI | accuracy: 40.3 -> 39.8 | Hurt |
| Expand IDEAL candidate prompts with automatically annotated examples (Auto-IDEAL) (ablation) | IDEAL · MRPC | accuracy: 65.2 -> 65.8 | Helped |
| Expand IDEAL candidate prompts with automatically annotated examples (Auto-IDEAL) (ablation) | IDEAL · SST-5 | accuracy: 49.4 -> 50.4 | Helped |
| Expand IDEAL candidate prompts with automatically annotated examples (Auto-IDEAL) (ablation) | IDEAL · DBpedia | accuracy: 90.8 -> 91.8 | Helped |
| Expand IDEAL candidate prompts with automatically annotated examples (Auto-IDEAL) (ablation) | IDEAL · RTE | accuracy: 57.4 -> 58.3 | Helped |
Surprising findings
- Random selection outperforms Vote-k in 3 of 18 cases, which the authors attribute to Vote-k selecting outliers and being biased.
- GPT-3.5-Turbo performs worse than GPT-Neo and GPT-J on classification prompts, attributed to its chat optimization.
Limitations
- Relies on the submodular condition, which is stated as holding for the influence function rather than fully proven in general.
- Main experiments use relatively small open models (GPT-J 6B, GPT-Neo 2.7B), with only limited tests on GPT-3.5-Turbo.
Experimental setup
Datasets: MRPC SST-5 MNLI DBpedia RTE HellaSwag MWoZ GeoQuery Xsum SST-2 BoolQ IMDb
Benchmarks: MRPC SST-5 MNLI DBpedia RTE HellaSwag MWoZ GeoQuery Xsum IMDb BoolQ Contrast Set
Metrics: accuracy joint accuracy test suite accuracy ROUGE-L time consumption
Baselines: Vote-k Random selection K-Means Maximizing facility location Fast Vote-k
Builds on: Vote-k influence maximization Sentence-BERT submodular greedy maximization
Closest papers by meaning
- Efficient Data Selection at Scale via Influence Distillation (NeurIPS 2025 · similarity 0.89)
- Harnessing Diversity for Important Data Selection in Pretraining Large Language Models (ICLR 2025 · similarity 0.88)
- MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models (NeurIPS 2024 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.87)
- MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.87)
- Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.