Research map / Large language models / Instruction and fine-tuning

Large Language Models are Demonstration Pre-Selectors for Themselves

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FEEDER, an LLM-tailored pre-selection of a small representative demonstration subset from training data (via sufficiency/necessity tree algorithm), which reduces data by over 20% for ICL and can be used in a bi-level fine-tuning loop.

Problem

Demonstration selection for in-context learning repeatedly retrieves from large training sets for each query, incurring high computational cost; selection is also not tailored to the specific LLM.

Approach

Pre-select a small subset of training data that is sufficient (plugging it in corrects LLM answers on other samples) and necessary (unplugging harms correct answers) for the specific frozen LLM, using a bottom-up tree approximation with one-shot LLM checks; the subset then serves as the candidate pool for existing selectors and as fixed fine-tuning data in a bi-level loop.

Key results

• FEEDER reduces the training pool by over 20% (paper reports retaining roughly half of samples) while keeping or improving ICL accuracy across GPT-2, GPT-neo, GPT-3, Gemma-2, Llama-2 and Llama-3.
• With FEEDER as pool, simple Similarity selection can match or beat Diversity selection on the full training set in many settings (e.g., Gemma-2 SUBJ 5-shot Similarity 94.5 vs 91.5 on full data).
• In the bi-level fine-tuning setting, GPT-2 SST-2 1-shot random accuracy rises from 61.3 on full training data to 89.5 with FEEDER subset.

Measured effects reported in the paper

Surprising findings

• Pre-selected subsets of roughly half the training data often outperform the full training pool, which the authors attribute to removing noisy and redundant shots.
• One-shot LLM sufficiency checks (K=1) and a single run (R=1) already give strong performance, with more rounds sometimes hurting.

Limitations

• Sufficiency is assumed transitive over sets, which is a strong hypothesis.
• Selected subsets depend on the LLM's own knowledge and may amplify its biases, as the impact statement notes.
• Evaluation on larger LLMs and real-world datasets is deferred to the appendix and future work.

Experimental setup

Datasets: SST-2 SST-5 COLA TREC SUBJ FPB GSM8K SMCALFlow GPQA

Benchmarks: SST-2 SST-5 COLA TREC SUBJ FPB GSM8K SMCALFlow GPQA

Metrics: accuracy

Baselines: Random selector Similarity selector Diversity selector (MMR) Uncertainty selector Clustering selector Latent selector

Builds on: similarity-based demonstration selection maximal marginal relevance core-set selection

Closest papers by meaning

• Comparative Analysis of Demonstration Selection Algorithms for In-Context Learning in Large Language Models (Student Abstract) (AAAI 2025 · similarity 0.92)
• Mixture of Demonstrations for In-Context Learning (NeurIPS 2024 · similarity 0.89)
• What Makes In-context Learning Effective for Mathematical Reasoning (ICML 2025 · similarity 0.89)
• Many-Shot In-Context Learning (NeurIPS 2024 · similarity 0.88)
• MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.88)
• From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation (ICLR 2025 · similarity 0.88)