Research map / Large language models / Retrieval and knowledge
ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReFusion, which fuses retrieval representations directly into transformer hidden states (reranker and ordered-mask ranking schemes, searched via bi-level optimization) instead of concatenating retrieved text, improving few-shot NLU on 15 NKI tasks with RoBERTa-large.
Problem
Retrieval-concatenation augmentation for non-knowledge-intensive NLU tasks greatly increases input length, raising attention compute and causing truncation, while naive representation fusion yields limited gains.
Approach
Fuse retrieved representations into hidden states of each layer rather than the input, with a learnable reranker and a per-dimension ordered mask over retrievals, and an adaptive retrieval integrator that picks the best ranking scheme per layer via differentiable bi-level search.
Key results
- ReFusion with RoBERTa-large averages 74.3 across 15 NKI tasks versus 71.8 for LM-BFF in the few-shot setting.
- On QNLI and SNLI, ReFusion reaches 73.0 and 80.6 versus LM-BFF's 64.5 and 77.2.
- Latency breakdown on CA-512 is 37.4 s per sentence on MPQA versus 20.2 s for ReFusion, with the baseline RoBERTa-L at 7.35 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LM-BFF with ReFusion | RoBERTa-large few-shot NKI tasks · Avg-all | accuracy: 71.8 -> 74.3 | Helped |
| Replace LM-BFF with ReFusion | RoBERTa-large · TREC | accuracy: 84.8 -> 90.3 | Helped |
| Add retrieval fusion with adaptive retrieval integrator and all ranking schemes (ablation) | RoBERTa-large · six-task ablation average | accuracy: 76.7 -> 82.3 | Helped |
| Add reranker-ranked retrieval representations without adaptive retrieval integrator (ablation) | RoBERTa-large · six-task ablation average | accuracy: 76.7 -> 78.2 | Helped |
| Add ordered-mask-ranked retrieval representations without adaptive retrieval integrator (ablation) | RoBERTa-large · six-task ablation average | accuracy: 76.7 -> 77.7 | Helped |
| Use inner-product rather than L2-norm retrieval metric (ablation) | ReFusion · six-task ablation average | accuracy: 82.3 -> 82.6 | Helped |
| Change retrieval count from k=1 to k=32 (ablation) | ReFusion · six-task ablation average | accuracy: 82.6 -> 83.8 | Helped |
| Increase retrieval count beyond k=64 (ablation) | ReFusion · NKI tasks | accuracy | Mixed |
| Retrieve based on hidden states rather than input texts (ablation) | ReFusion · MPQA | accuracy: 87.9 -> 88.7 | Helped |
| Retrieve based on hidden states rather than input texts (ablation) | ReFusion · SUBJ | accuracy: 92.1 -> 91.7 | Hurt |
| Retrieve based on hidden states rather than input texts (ablation) | ReFusion · TREC | accuracy: 84.0 -> 85.6 | Helped |
| Use ReFusion instead of T5-base | T5-base few-shot NKI tasks · Avg-all | accuracy: 78.2 -> 79.6 | Helped |
Surprising findings
- Retrieval-representation fusion with plain addition (RF-add) achieves higher accuracy than 16-retrieval concatenation while keeping FLOPs nearly unchanged.
- The gains from ReFusion are smaller under LoRA fine-tuning (63.8 to 64.7 average) than under full fine-tuning.
Limitations
- Retrieval during inference (especially 'Hidden' queries) adds substantial latency, e.g., 108 s per sentence on MPQA.
- Search space grows as 9^N across layers; search is costly and the paper does not fully characterize its cost.
- Hyperparameters were not grid-searched, and results on the full training set are reported only on a subset of tasks.
Experimental setup
Datasets: SST-2 SST-5 MR CR MPQA SUBJ TREC CoLA MNLI MNLI-m SNLI QNLI
Benchmarks: SST-2 SST-5 MR CR MPQA SUBJ TREC CoLA MNLI MNLI-m SNLI QNLI
Metrics: accuracy standard deviation across 5 seeds inference latency (seconds per sentence) FLOPs
Baselines: LM-BFF DART KPT CA-512 T5-base Splinter FewshotBART
Builds on: LM-BFF nested dropout Gumbel softmax DARTS-style bi-level optimization RoBERTa
Closest papers by meaning
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.86)
- Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation (AAAI 2025 · similarity 0.85)
- FreeRet: MLLMs as Training-Free Retrievers (ICML 2026 · similarity 0.84)
- BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models (ICLR 2024 · similarity 0.84)
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.84)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.