Research map / Large language models / Retrieval and knowledge

Supervised Knowledge Makes Large Language Models Better In-context Learners

ICLR 2024 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SuperContext, which injects a fine-tuned small discriminative model's prediction and confidence into LLM prompts to improve OOD NLU generalization and reduce hallucination on SQuAD 2.0.

Problem

LLMs underperform task-specific small models on out-of-distribution NLU classification and hallucinate on unanswerable QA; existing work treats LLMs and SLMs as separate paradigms.

Approach

Insert a fine-tuned discriminative model's (ELECTRA-large / RoBERTa-large) predicted label and confidence into the LLM's in-context prompt, so the LLM can reason over supervised task knowledge plus its own domain knowledge, with an optional interpreter prompt asking for influential examples and rationale.

Key results

• SuperContext with ChatGPT achieves 80.05% average OOD accuracy across 15 GLUE-X datasets, versus 79.86% for ELECTRA-large and 66.67% for zero-shot ChatGPT.
• SuperContext (16-shot) with Llama2-7B-chat reaches 79.08% average, up from 58.54% for 16-shot Llama2-chat.
• On SQuAD 2.0 with ChatGPT, no-answer accuracy rises from 32.35% to 54.65% under SuperContext (zero-shot).

Measured effects reported in the paper

Surprising findings

• Llama2-7B-chat with SuperContext in 16-shot setting beats zero-shot SuperContext on ChatGPT on several OOD tasks.
• Adding the interpreter prompt gave little benefit over plain SuperContext on most tasks, possibly due to information overload.

Limitations

• Only ChatGPT (gpt-3.5-turbo) and Llama2-7B-chat tested; other LLMs such as GPT-4 not analyzed.
• Evaluation limited to NLU classification and a single QA dataset; generation tasks left for future work.
• Interpreter rationales are LLM-generated and may hallucinate, so analysis is only indicative.

Experimental setup

Datasets: GLUE-X SQuAD 2.0 GLUE

Benchmarks: GLUE-X SQuAD 2.0

Metrics: accuracy F1 exact match valid JSON rate accuracy on no-answer questions accuracy on has-answer questions

Baselines: 16-shot ICL BM25 in-context example selection ELECTRA-large cluster+filter fine-tuned Llama2-7B-chat (single-turn and multi-turn)

Builds on: GLUE-X contrastive decoding SuperICL

Closest papers by meaning

• Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.91)
• The Representation Landscape of Few-Shot Learning and Fine-Tuning in Large Language Models (NeurIPS 2024 · similarity 0.90)
• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.90)
• TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.90)
• Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.90)
• A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.90)