Research map / Large language models / Instruction and fine-tuning

ELICIT: LLM Augmentation Via External In-context Capability

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ELICIT, which stores in-context task vectors per task and model in an external capability library, retrieves them with a trained SimCSE-based classifier, and adds them to hidden states at a pre-selected layer to elicit capabilities without extra inference tokens.

Problem

Adapting LLMs to new tasks efficiently: fine-tuning is costly and in-context learning needs demonstrations and extra tokens per inference.

Approach

Build an external library of task vectors (hidden states from 16-shot ICL prompts) with per-task optimal intervention layers, retrieve the relevant vector for a query using a trained binary retriever with threshold filtering, and add it to the hidden state at inference so that zero-shot queries behave like ICL without extra tokens.

Key results

• Across 20 tasks and 4 models, ELICIT averages 11.4% improvement over zero-shot with the same token usage (Llama3 avg 31.2 to 43.5).
• With a math-only library on Mistral, Math accuracy rises from 2.6% to 21.3% while other domains stay roughly unchanged.
• Combining ELICIT with BM25 raises Pythia average from 22.1% to 28.3%.
• The trained retriever reaches AUC 0.96 versus at most 0.28 for cosine, Euclidean, or t-SNE similarity baselines on Llama3.

Measured effects reported in the paper

Surprising findings

• Combining ELICIT with BM25 degrades performance for larger models (Llama3 and Mistral average drops, e.g., Llama3 55.8 to 53.1), while it helps smaller models.
• Higher intervention strength improves task accuracy but degrades language modeling cross-entropy, showing a trade-off.

Limitations

• Zero-shot accuracy of base models with the task template is near zero in the pure ICL setting, so the evaluation relies on added task templates.
• Gains are mixed for larger models when combined with BM25 retrieval.
• Requires building a per-model library and a per-model optimal layer, plus a trained retriever.

Experimental setup

Datasets: CommonsenseQA OpenBookQA HellaSwag BoolQ Big-Bench Hard ARC-Challenge MathQA MMLU Pro-MATH Crows-Pairs BBQ-Age Ethics-Commonsense Ethics-Justice

Benchmarks: CommonsenseQA OpenBookQA HellaSwag BoolQ Big-Bench Hard ARC-Challenge MathQA MMLU Pro-MATH Crows-Pairs BBQ-Age Ethics-Commonsense Ethics-Justice

Metrics: accuracy precision-recall AUC cross-entropy loss on WikiText

Baselines: Zero-shot 16-shot ICL BM25 retrieval (16-shot)

Builds on: task vectors in-context learning BM25 SimCSE

Closest papers by meaning

• LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.89)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.89)
• Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.88)
• Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.88)
• EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.88)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models (NeurIPS 2024 · similarity 0.88)