Research map / Large language models / Retrieval and knowledge

WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia

NeurIPS 2024 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WikiContradict, a 253-instance human-annotated benchmark of real-world inter-context conflicts from Wikipedia maintenance tags, evaluates LLMs in RAG with contradictory passages, and proposes an LLM-judge (WikiContradictEval) built on Llama-3-70b.

Problem

LLMs' handling of real-world inter-context knowledge conflicts, where retrieved passages from the same trusted source (Wikipedia) are equally credible and contradict each other, is poorly studied; prior conflict benchmarks are synthetic and focus on context-memory conflicts.

Approach

Builds a QA benchmark from Wikipedia articles tagged inconsistent / self-contradictory / contradict-other by editors, annotating two contradictory passages, explicit/implicit conflict types, and answers per passage, then evaluates models under five prompt templates (no context, single passage, two contradictory passages) with human judgement and a few-shot Llama-3-70b judge.

Key results

• Llama-3-70b-instruct's correct rate on the two-passage prompt rose from 10.4% (template 4) to 43.8% (template 5) when told to consider conflicting context.
• WikiContradictEval with GPT-4 as judge reached F-score 82.5 on correct responses (Llama-3-70b judge F-score 80.4), with the open-source Llama-3-70b judge used for the automatic evaluation.
• Under the Llama-3-70b judge, Mistral-7b-inst (50.6%) and Llama-3-70b-inst (45.8%) had the highest correct rates on template 5 across all 253 instances, though the authors note Mistral is likely overestimated.

Measured effects reported in the paper

Surprising findings

• GPT-4 often relied on its internal knowledge over the given context (stubborn behavior), scoring 87.8% on template 3.
• Judge models and humans diverged for Mistral-7b-inst, which often gave both answers while explaining one was wrong, leading the LLM judge to overestimate its correctness.

Limitations

• Benchmark is English-only and limited to 253 instances.
• Reliance on Wikipedia maintenance tags may skew coverage toward certain contradiction types.
• Only text-text contradictions are covered; cross-modal contradictions are not.

Experimental setup

Datasets: Wikipedia articles with content issues (maintenance tags)

Benchmarks: WikiContradict WikiContradict_HumanEval

Metrics: accuracy precision recall F-score macro-F Cohen's kappa

Baselines: Mistral-7b-instruct Mixtral-8x7b-instruct Llama-2-70b-chat Llama-3-70b-instruct GPT-4

Builds on: FreshLLM

Closest papers by meaning

• ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence (NeurIPS 2024 · similarity 0.91)
• $\texttt{ConflictBank}$: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMs (NeurIPS 2024 · similarity 0.90)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.89)
• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.89)
• RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.88)
• Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (ICLR 2024 · similarity 0.88)