Research map / Large language models / Instruction and fine-tuning

Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SIFT, a transductive active-learning data selection method that minimizes response uncertainty about a prompt for test-time fine-tuning of LLMs, outperforming nearest-neighbor retrieval on the Pile and enabling adaptive compute stopping.

Problem

Selecting fine-tuning data for test-time, prompt-specific fine-tuning of LLMs; standard nearest-neighbor retrieval selects redundant data and can hurt performance when information is duplicated.

Approach

Select data sequentially to minimize the surrogate-model posterior uncertainty about the response to the prompt (transductive active learning with a kernel/linear surrogate), which accounts for redundancy among selected points and thus unifies retrieval with active learning; uncertainty estimates also predict fine-tuning gains and enable adaptive stopping.

Key results

• On the Pile, SIFT reaches 73.5% of base-model bits per byte versus 78.3% for NN and 133.3% for the NN failure mode (GPT-2, relative to base, lower is better).
• Test-time fine-tuned Llama-3.2 (3B) with SIFT achieves 0.557 bits per byte on the Pile, and Phi-3 (3.8B) achieves 0.595, reported as new state of the art on the Pile benchmark excluding copyrighted datasets.
• The normalized uncertainty estimate correlates at about 0.4 with bits per byte, and the denormalized estimate at about 0.5 or more, enabling an adaptive stopping rule with compute-proportional gains.

Measured effects reported in the paper

Surprising findings

• Nearest neighbor repeatedly selecting the closest point (NN-F) performs worse than the unadapted base model on the Pile.
• Test-time fine-tuning generally outperforms or matches in-context learning with the same selected data, even though in-context learning uses the same data in the context window.

Limitations

• Evaluation is limited to perplexity/language modeling on the Pile; broader non-perplexity tasks are left to future work.
• Uncertainty is computed with a surrogate linear model over normalized RoBERTa embeddings, and the theory relies on assumptions such as submodularity of uncertainty reduction.
• Evaluation uses only 1% of the Pile test set (0.1% for Phi-3), and Phi-3 uses LoRA rather than full fine-tuning.

Experimental setup

Datasets: The Pile training set (210M sequences) as data space The Pile test set (1% subset)

Benchmarks: The Pile test set (per-subset bits per byte)

Metrics: bits per byte relative bits per byte vs base model standard error correlation of uncertainty with bits per byte

Baselines: Nearest Neighbor retrieval (NN) NN failure mode (NN-F) Uncertainty Sampling (US) In-context learning Phi-3 14B Gemma-2 27B

Builds on: test-time training transductive active learning dynamic evaluation Faiss nearest-neighbor retrieval

Closest papers by meaning

• Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.89)
• Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.89)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.88)
• Scaling Up Active Testing to Large Language Models (NeurIPS 2025 · similarity 0.88)
• DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.88)
• Efficient Evaluation of LLM Performance with Statistical Guarantees (NeurIPS 2026 · similarity 0.88)