Research map / Large language models / Instruction and fine-tuning

Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Reference Trustable Decoding (RTD), a training-free method that builds a datastore of hidden states from training examples and fuses a kNN-style reference distribution with the LM_Head output at decoding time.

Problem

Adapting LLMs to downstream tasks usually needs either in-context examples, which lengthen inputs and KV-cache, or fine-tuning, which needs gradient computation and substantial memory.

Approach

Bypass the LM_Head with a retrieval-based reference distribution: a datastore of last-token hidden states paired with labels from task training data is queried by Euclidean distance, and the resulting distribution is interpolated with the model's softmax output. A multi-head variant splits hidden states into per-head sub-datastores and averages their reference distributions.

Key results

• Average accuracy across the language-understanding benchmarks rises from 56.41 (zero-shot baseline) to 63.31 with RTD and 68.88 with 5-shot RTD, versus 65.28 for 5-shot ICL.
• On LLaMA2-7B, RTD raises ARC (E&C) from 30.1 to 41.4 and PIQA from 56.5 to 71.4 without adding input tokens.
• On Wikipedia knowledge injection with LLaMA2-7B-Chat, Wiki RTD reaches 44.4% accuracy at 46.5 ms latency, versus 39.0% baseline and 29.0% for Wiki RAG (>200 ms).

Measured effects reported in the paper

Surprising findings

• Naive Wikipedia RAG scored below the zero-shot baseline (29.0 vs 39.0), which the authors attribute to sequence overflow past LLaMA2-7B-Chat's 4096-token pretraining length.
• RTD performed worse than the zero-shot baseline on some settings, e.g. MPT-7B OpenbookQA (27.2 vs 29.4) and LLaMA3-8B MMLU 5-shot RTD underperformed 5-shot ICL.

Limitations

• Large reference datastores, especially for generative tasks, are hard to compress and may be impractical.
• Several hyperparameters (datastore size, k, temperature T, lambda) need tuning, and the best lambda varies by task.
• Results are reported as single deterministic runs with no error bars, and the paper reports RTD still trails full fine-tuning on MMLU.

Experimental setup

Datasets: MMLU ARC PIQA OpenBookQA C-MMLU MMLU-Recall Tiny-Shakespeare Wikipedia C-Eval validation Belebele zho_Hans ACLUE

Benchmarks: MMLU ARC PIQA OpenBookQA C-MMLU Tiny-Shakespeare perplexity

Metrics: accuracy perplexity confused rate tokens per second extra memory usage

Baselines: 5-shot ICL LoRA full-parameter fine-tuning Wiki RAG zero-shot baseline

Builds on: retrieval-augmented generation LoRA in-context learning faiss

Closest papers by meaning

• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
• Adaptive Draft-Verification for Efficient Large Language Model Decoding (AAAI 2025 · similarity 0.88)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.88)
• Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models (NeurIPS 2025 · similarity 0.88)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
• ELICIT: LLM Augmentation Via External In-context Capability (ICLR 2025 · similarity 0.88)