Research map / Large language models / Language, safety and interpretability
GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains encoder-decoder gist models with an attention bottleneck so that a few gist-token embeddings score and select in-context examples via GistScore, either per dataset or via one multi-task model usable out-of-the-box.
Problem
Choosing in-context examples for LLMs strongly affects ICL performance, but off-the-shelf retrievers are suboptimal, trained selectors need task data or LLM feedback, and BERTScore-Recall is quadratic and costly for long text.
Approach
Adapt Instruction Gisting to example encoding: fine-tune encoder-decoder models so the output can only attend to a few gist tokens over the input, then score candidates by a BERTScore-Recall-style similarity over those few gist embeddings (GistScore), with a multi-task variant that gists instructions plus inputs for training-free use.
Key results
- Fine-tuned GistScore (GS[F], 1 token) beats SBERT and BSR by up to 21 and 11 points and the trained baselines CEIL and LLM-R by 5 and 8 points on average.
- Multi-task GS[M] recovers much of the fine-tuned gain, matches or beats trained baselines EPR, CEIL and LLM-R, and generalizes to held-out datasets, domains, and languages.
- GistScore selection is up to four orders of magnitude faster than BSR and three orders faster than BM25 on the stated timings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace off-the-shelf retrieval with dataset-fine-tuned GistScore | ICL example selection · 21 ICL datasets | ICL performance: over 20% absolute gain over off-the-shelf retrievers; 5% over the best prior methods | Helped |
| Use multi-task-trained GistScore without task-specific fine-tuning | ICL example selection · New tasks and datasets | ICL performance | Helped |
| Use set-selection with 15 gist tokens instead of independent ranking with a single token (ablation) | GistScore for Semantic Parsing · Semantic Parsing | Exact-Match Accuracy: 15 tokens | Helped |
| Use set-selection with 15 gist tokens (ablation) | GS[Multitask] · Semantic Parsing | Exact-Match Accuracy: 15 tokens | Helped |
| Use more gist tokens without set-selection (ablation) | GS[Finetune] · Semantic Parsing | Exact-Match Accuracy | Helped |
| Use more gist tokens without set-selection (ablation) | GS[Multitask] · Semantic Parsing | Exact-Match Accuracy | No clear effect |
| Use set-selection rather than independent ranking (ablation) | GistScore for Semantic Parsing · Semantic Parsing | Exact-Match Accuracy | Helped |
| Use a single gist token rather than additional gist tokens (ablation) | GistScore · Most datasets | ICL performance | Helped |
| Use 2 selected examples instead of 8 (ablation) | GistScore selection · ICL with GPT-Neo-2.7B | ICL performance: just 2 examples | Helped |
| Use single-token GistScore instead of BSR | Training-free example selection · ICL example selection | selection time: up to four orders of magnitude faster | Helped |
Surprising findings
- GistScore-based selection can exceed the underlying gist model's own accuracy, especially on compositional and GSM8K tasks where the gist model itself fails.
- Only 2 GistScore-selected examples can outperform 8-shot ICL with SBERT and BM25 retrievers.
Limitations
- Semantic parsing needs extra gist tokens and set-selection; a single gist token is insufficient there.
- Multi-task gist model excludes semantic parsing from training, so its generalization to that task relies on set-selection and additional tokens.
- Evaluated mainly on base LLMs up to 13B and OpenAI base models; gains from stronger inference LLMs are not fully characterized.
Experimental setup
Datasets: FLAN 2022 (subset, ~5M prompts) MNLI QNLI RTE MedNLI WANLI XNLI GSM8K AGNews SST5 SST2 Rotten Tomatoes
Benchmarks: MNLI QNLI RTE MedNLI WANLI XNLI GSM8K AGNews SST5 SST2 Rotten Tomatoes MRPC
Metrics: accuracy exact match ROUGE-L
Baselines: SBERT BM25 BSR Set-BSR EPR CEIL LLM-R Random
Builds on: Gisting BERTScore-Recall EPR CEIL LLM-R
Closest papers by meaning
- Implicit In-context Learning (ICLR 2025 · similarity 0.87)
- Capturing In-Context Learning Dynamics with Task Operators (NeurIPS 2026 · similarity 0.87)
- In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.87)
- MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning (ICML 2025 · similarity 0.86)
- Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · similarity 0.86)
- Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.