Research map / Large language models / Retrieval and knowledge
To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks LLM over-reliance on incorrect external contexts across QA sets (incl. new Reddit-sourced RedditQA) and proposes self-guided (SCR) and rule-based (RCR) confidence reasoning, plus CR-DPO fine-tuning to improve situated faithfulness.
Problem
LLMs augmented with external context (e.g., RAG) over-rely on that context even when it is incorrect or misleading, conflicting with their internal knowledge and reducing answer accuracy.
Approach
Frame the problem as situated faithfulness: the model should calibrate trust in external context against its own confidence in internal knowledge. Self-guided confidence reasoning lets the LLM reason about both sources in text (implicitly or via CoT), while rule-based methods extract confidence signals and apply fixed decision rules; CR-DPO trains the model with DPO on self-sampled chosen/rejected
Key results
- SCR improves over direct input augmentation by +17.8 (GPT-4o mini) and +24.2 (GPT-4o) overall situated faithfulness.
- For Llama-3-8B, CR-DPO improves ExplicitSCR by 8.9% on average (in-distribution +10.4%, out-of-distribution +5.9%).
- For Llama-3-8B, InternalConf outperforms SCR (e.g., Acc_f on TriviaQA 64.6% vs 34.0% for ExplicitSCR), showing RCR is stronger for the weaker model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ImplicitSCR instead of DIA | GPT-4o mini · Total | overall situated faithfulness: +17.8 | Helped |
| Use SCR instead of DIA | GPT-4o · Total | overall situated faithfulness: +24.2 | Helped |
| Use CR-DPO to fine-tune ExplicitSCR | Llama-3-8B · Total | overall situated faithfulness: +8.9% | Helped |
| Use CR-DPO rather than untrained ExplicitSCR | Llama-3-8B · TriviaQA | overall situated faithfulness: 60.4 -> 73.2 | Helped |
| Use CR-DPO rather than untrained ExplicitSCR | Llama-3-8B · RedditQA | overall situated faithfulness: 65.1 -> 79.6 | Helped |
| Remove explicit CoT confidence reasoning (ablation) | CR-DPO-ST · NaturalQA | overall situated faithfulness: 54.9 -> 38.7 (-16.2) | Hurt |
| Replace DPO with supervised fine-tuning (ablation) | CR-DPO-ST · TriviaQA | overall situated faithfulness: 68.4 -> 59.4(-9.0) | Hurt |
| Remove RedditQA from training tasks (ablation) | CR-DPO-ST · FreshQA | overall situated faithfulness: 53.4 -> 50.9(-2.5) | Hurt |
| Train on TriviaQA only (ablation) | CR-DPO-ST · RedditQA | overall situated faithfulness: 76.7 -> 69.3(-7.4) | Hurt |
| Use GPT-4o-sourced reasoning paths instead of self-sampled paths (ablation) | CR-DPO-ST trained on TriviaQA only · RedditQA | overall situated faithfulness: 69.3 -> 67.9(-8.8) | Hurt |
| Use two chosen/rejected reasoning pairs instead of one (ablation) | CR-DPO · in-distribution datasets | overall situated faithfulness: +2.2% | Helped |
| Use two chosen/rejected reasoning pairs instead of one (ablation) | CR-DPO · out-of-distribution datasets | overall situated faithfulness: -0.5% | Hurt |
Surprising findings
- Learning from self-sampled reasoning paths outperforms reasoning paths sampled from the stronger GPT-4o when fine-tuning Llama-3-8B.
- Better-calibrated confidence scores (via isotonic regression) do not necessarily improve situated faithfulness.
Limitations
- Evaluation limited to short-form QA with a single context at a time; multi-context RAG and long-form QA not studied.
- CR-DPO adds inference cost from chain-of-thought reasoning.
- RedditQA training data is not human-verified and thus noisy; the RedditQA test set is small (176 samples).
Experimental setup
Datasets: TriviaQA NaturalQA PopQA ConflictQA FreshQA ClashEval RedditQA
Benchmarks: RedditQA FreshQA ClashEval TriviaQA PopQA NaturalQA
Metrics: accuracy given true contexts (Acc_t) accuracy given false contexts (Acc_f) overall situated faithfulness (SF) expected calibration error AUC-ROC
Baselines: Direct input augmentation TACS (LR) TPC InternalConf ContextConf InternalEval ContextEval ActiveRAG
Builds on: direct preference optimization ActiveRAG Truth-Aware Context Selection ClashEval ConflictQA
Closest papers by meaning
- Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.91)
- FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows" (ICLR 2025 · similarity 0.91)
- Information Discernment in Large Language Models (NeurIPS 2026 · similarity 0.90)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.90)
- HonestLLM: Toward an Honest and Helpful Large Language Model (NeurIPS 2024 · similarity 0.90)
- ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.