Research map / Large language models / Retrieval and knowledge
Decoupled Context Processing for Context Augmented Language Modeling
Zonglin Li, Ruiqi Guo, Sanjiv Kumar
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add retrieved context to the language model (ablation) | Decoupled Encoder-Decoder (base, large, XL) · C4 validation | bits-per-byte | Improved |
| use decoupled encoder-decoder cross attention instead of Retro chunked cross attention | Retro · C4 validation | bits-per-byte | Improved |
| use trained dense retriever instead of BM25 (ablation) | Decoupled Encoder-Decoder · C4 validation | bits-per-byte | No clear effect |
| train from scratch instead of fine-tuning pre-trained models (ablation) | mT5 encoder-decoder · C4 validation | bits-per-byte | No clear effect |
| add 100,000 extra steps with lower learning rate 2e-4 (ablation) | Decoupled Encoder-Decoder · C4 validation | bits-per-byte | Improved |
| warm start with 100,000 steps on top 10% context-utility data (ablation) | Large and XL models · C4 training | training accuracy | Improved |
| grounded context transfer versus non-grounded outputs | Decoupled Encoder-Decoder · Natural Question | exact match accuracy: 51.5 -> 35.4 | Improved |
| remove passages containing the original output and rerun inference (ablation) | Decoupled Encoder-Decoder · Natural Question | fraction of outputs changed but still grounded: 2,765 out of 3,354 | Mixed |
| precompute and cache context encodings instead of encoding at inference | Decoupled Encoder-Decoder XL · Natural Question latency | retrieval time (ms): 200ms -> 66ms | Improved |
Closest papers by meaning
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.89)
- Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.88)
- Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models (NeurIPS 2024 · similarity 0.87)
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.87)
- Provence: efficient and robust context pruning for retrieval-augmented generation (ICLR 2025 · similarity 0.87)
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
