Research map / Large language models / Retrieval and knowledge
$\texttt{ConflictBank}$: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CONFLICTBANK, a large benchmark of 7.45M claim-evidence pairs and 553k QA pairs covering misinformation, temporal, and semantic knowledge conflicts, and uses it to study retrieved and embedded conflicts in 12 LLMs.
Problem
Knowledge conflicts, a major source of LLM hallucinations, have been studied mainly between parametric knowledge and retrieved context; conflicts within models' encoded knowledge and their interplay remain under-explored.
Approach
Builds conflict claims by substituting Wikidata facts along three causes (misinformation, temporal, semantic), generates styled evidence with an LLM, filters by refusal features, NLI entailment and conflict confirmation, and uses the resulting data to probe retrieved conflicts, embedded conflicts via continued pre-training, and their interplay.
Key results
- Across all four model series, models show memorization ratios below 50% when given only contradictory external evidence, but above 50% when parametric-consistent evidence is also present.
- Semantic conflicts yield the lowest memorization ratio in retrieved-conflict settings; e.g., LLaMA2-70B MR is 3.73% for semantic vs 9.45% for misinformation with two evidence pieces.
- In continued pre-training, LLaMA2-7B/Mistral-7B OAR drops by 41.5% and 37.6% when only one-third of data is conflicting (2:1 ratio), while prepending default evidence restores default answers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide only contradictory external evidence instead of also providing evidence matching the model's parametric knowledge | LLMs evaluated on retrieved conflicts · Retrieved knowledge conflicts | MR | Mixed |
| Use temporal or semantic conflict evidence instead of misinformation evidence | LLMs evaluated on retrieved conflicts · Retrieved knowledge conflicts | MR: 7.77% & 3.73% v.s. 9.45% | Hurt |
| Increase parameter size within a model series | LLMs evaluated with two pieces of external evidence · Retrieved knowledge conflicts | MR | Hurt |
| Add conflicting examples to continued pre-training data | LLaMA3-8B, Qwen1.5-4B, and Mistral-7B · Embedded knowledge conflicts, answering without evidence | OAR: 54.43 -> 36.53 -> 34.23 -> 32.70; 70.83 -> 66.65 -> 65.39 -> 61.03; 58.29 -> 36.38 -> 31.66 -> 29.84 | Hurt |
| Add conflicting examples to continued pre-training data | LLaMA3-8B, Qwen1.5-4B, and Mistral-7B · Embedded knowledge conflicts, answering with default evidence | OAR: 99.67, 99.33, 99.47, 99.73; 99.82, 99.82, 99.82, 99.88; 99.52, 99.15, 93.77, 94.98 | No clear effect |
| Add descriptions of temporal or semantic conflicts to the question (ablation) | LLMs evaluated with two pieces of external conflict evidence · Retrieved knowledge conflicts | MR: 67.52% to 8.31%; 55.37% to 41.80% | Hurt |
| Place conflicting evidence later in the evidence sequence | LLaMA2 models · Retrieved knowledge conflicts | MR: 56.44% ... 38.97% | Hurt |
| Increase the proportion of embedded conflicting data from 1:0 to 2:1, 1:1, or 2:3 (ablation) | LLaMA3-8B, Qwen1.5-4B, and Mistral-7B · Embedded knowledge conflicts, answering without evidence | OAR: 54.43 -> 36.53 -> 34.23 -> 32.70 | Hurt |
Surprising findings
- Models are more sensitive to implicit temporal and semantic conflicts than to explicit misinformation, which seems counter-intuitive.
- Adding descriptive context to questions sharply reduces memorization ratio (e.g., LLaMA3-70B from 67.52% to 8.31%).
Limitations
- Conflict pairs are generated by LLMs rather than mined from real pre-training corpora.
- Evidence filtering relies on automatic NLI and classifier checks with only manual spot checks of 200 examples.
Experimental setup
Datasets: Wikidata (April 2024 dump)
Benchmarks: ConflictBank
Metrics: memorization ratio original answer ratio counter answer ratio
Baselines: GPT-4o
Builds on: Xie et al. 2024 conflict evidence generation Longpre et al. 2021 entity substitution Gekhman et al. 2024
Closest papers by meaning
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR 2024 · similarity 0.91)
- WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia (NeurIPS 2024 · similarity 0.90)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.89)
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models (AAAI 2026 · similarity 0.89)
- KScope: A Framework for Characterizing the Knowledge Status of Language Models (NeurIPS 2025 · similarity 0.89)
- PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.